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Preface 


The  Office  of  Force  Transformation  (OFT)  and  the  Office  of  the 
Assistant  Secretary  of  Defense  for  Networks  and  Information  Integra¬ 
tion  (OASD  [Nil])  have  developed  a  conceptual  framework  for  con¬ 
ducting  analyses  and  enhancing  understanding  of  network-centric 
operations  (NCO)  capabilities.  RAND  is  one  of  the  supporting  orga¬ 
nizations  that  assisted  the  Office  of  the  Secretary  of  Defense  in  devel¬ 
oping  the  NCO  Conceptual  Framework  (NCO  CF).  The  NCO  CF 
has  several  objectives:  to  provide  a  better  understanding  of  key  NCO 
attributes  and  their  interrelationships;  to  provide  metrics  to  measure 
progress  in  developing  transformed,  network-centric  forces;  and  to 
help  understand  and  articulate  how  NCO  capabilities  can  be  a  source 
of  combat  power. 

RAND  has  applied  the  NCO  CF  to  the  air-to-air  combat  mis¬ 
sion.  We  used  the  framework  to  examine  the  results  of  the  Joint  Tac¬ 
tical  Information  Distribution  System  (JTIDS)  Operational  Special 
Project.  This  project  examined  the  performance  of  tactical  fighter  air¬ 
craft  (F- 1 5s)  equipped  with  Link  16  data  communications  terminals 
and  found  that  F-15s  equipped  with  Link  16  were  significantly  more 
effective  in  air  combat  than  F-15s  equipped  with  only  voice  commu¬ 
nications. 

This  report  describes  the  results  of  the  case  study  that  involved 
analyzing  the  capabilities  of  Link  16  data  and  voice  communications 
networks,  conducting  interviews  with  experienced  fighter  pilots,  and 
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building  a  quantitative  model  to  calculate  NCO  CF  metrics  for  mis¬ 
sion  capability  packages  designed  for  the  air-to-air  combat  mission. 

This  case  study  provides  useful  insights  into  the  application  of 
the  NCO  CF  and  associated  metrics.  The  report  highlights  the 
advantages  NCO  capabilities  can  potentially  provide  U.S.  air  forces 
in  the  air  superiority  mission.  It  also  demonstrates  the  feasibility  of 
applying  the  NCO  CF  in  a  quantitative  fashion  to  the  chain  of  infer¬ 
ences  contained  in  the  network-centric  warfare  hypothesis.  This 
report  should  be  of  use  as  a  starting  point  for  those  seeking  to  use  the 
NCO  CF  to  analyze  the  impact  of  NCO  capabilities  in  more  compli¬ 
cated  military  mission  areas. 

This  research  was  conducted  for  OFT  within  the  Acquisition 
and  Technology  Policy  Center  of  the  RAND  National  Defense 
Research  Institute  (NDRI).  NDRI  is  a  federally  funded  research  and 
development  center  sponsored  by  the  Office  of  the  Secretary  of 
Defense,  the  Joint  Staff,  the  unified  commands,  and  the  defense 
agencies. 

For  more  information  on  RAND’s  Acquisition  and  Technology 
Policy  Center,  contact  the  Director,  Philip  Anton.  Fie  can  be  reached 
by  e-mail  at  ATPC_Director@rand.org;  by  phone  at  310-393-0411, 
extension  7798;  or  by  mail  at  RAND,  1776  Main  Street,  Santa  Mon¬ 
ica,  CA,  90407-2138.  More  information  about  RAND  is  available  at 
www.rand.org. 
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Summary 


In  the  mid-1990s,  the  U.S.  Air  Force  at  the  request  of  Congress  con¬ 
ducted  the  Joint  Tactical  Information  Distribution  System  (JTIDS) 
Operational  Special  Project.  In  this  exercise,  the  capabilities  of  F-15 
air  superiority  aircraft  equipped  with  voice-only  communications 
were  compared  with  F-15s  equipped  with  voice  and  JTIDS  Link  16 
data  link  communications  in  tactical  air-to-air  combat.  More  than 
12,000  sorties  were  flown  in  this  special  project.  Blue  offensive  coun¬ 
terair  packages  composed  of  these  F-15s  ranged  in  size  from  two  to 
eight  aircraft.  In  all  cases,  the  packages  were  controlled  and  cued  by 
Airborne  Warning  and  Control  System  (AWACS)  aircraft.  The  size 
of  the  engagements  ranged  from  two  Blue  fighters  on  two  Red  fight¬ 
ers  to  eight  Blue  fighters  on  16  Red  fighters.  Engagements  occurred 
during  daylight  and  night  conditions.  The  primary  independent 
variable  was  whether  the  Blue  F-15s  were  equipped  with  the  Link  16 
data  link  or  with  conventional  voice  communications  only.  The 
capability  of  the  Red  aircraft  remained  consistent  during  the  project. 

On  average,  Blue  offensive  counterair  packages  equipped  with 
Link  16  achieved  a  two-and-a-half  times  improvement  in  kill  ratio 
(Red  aircraft  to  Blue  aircraft  “destroyed”),  both  during  the  day  and  at 
night.  However,  it  was  unclear  how  and  why  this  significant  improve¬ 
ment  in  force  effectiveness  arose.  The  aim  of  this  study  is  to  under¬ 
stand  whether  this  increase  in  combat  effectiveness  stemmed  from  the 
network-centric  capabilities  of  F-15  aircraft  equipped  with  Link  16 
and  fighter  pilots  able  to  effectively  use  data  link  communications. 
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The  original  Network-Centric  Warfare  (NCW)  hypothesis  pos¬ 
its  the  following  relationships  between  twenty-first  century  informa¬ 
tion  technologies,  information  sharing,  and  warfighting  capabilities: 

•  “A  robustly  networked  force  improves  information  sharing 

•  “Information  sharing  enhances  the  quality  of  information  and 
shared  situational  awareness 

•  “Shared  situational  awareness  enables  self-synchronization,  and 
enhances  sustainability  and  speed  of  command 

•  “These,  in  turn,  dramatically  increase  mission  effectiveness.” 
(Alberts  and  Garstka,  2001.) 

The  Network-Centric  Operations  Conceptual  Framework 
(NCO  CF),  developed  by  Office  of  Force  Transformation  (OFT) 
and  the  Office  of  the  Assistant  Secretary  of  Defense  for  Networks  and 
Information  Integration  (OASD  [Nil]),  provides  a  more  detailed  and 
precise  elaboration  of  the  NCW  hypotheses.1  It  includes  NCO  capa¬ 
bility  concepts  (such  as  the  degree  of  networking,  degree  of  informa¬ 
tion  sharing,  and  situational  awareness)  and  hypotheses  for  how  these 
concepts  relate  to  and  influence  each  other.  The  result  is  an  inter¬ 
linked  set  of  NCO  capabilities  that  describe  how  they  in  combination 
can  lead  to  improvements  in  overall  military  force  effectiveness. 
Importantly,  the  NCO  CF  describes  subsidiary  attributes  and  metrics 
for  assessing  NCO  capability  concepts,  making  it  possible  to  deter¬ 
mine  whether  and  how  possession  of  a  particular  NCO  capability 
relates  to  improvements  in  force  effectiveness.  Figure  S.l  shows  a  top- 
level  view  of  the  NCO  CF,  including  its  top-level  NCO  capability 
concepts  and  the  hypothesized  interactions  between  them.2  All  link¬ 
ages  reflect  positive  relationships;  for  example,  it  is  hypothesized  that 


1  Note  that  we  use  the  terms  NCO  and  NCW  interchangeably  in  this  report. 

2  The  NCO  CF  is  described  in  Signori  et  al.  (2002);  Evidence  Based  Research,  Inc.  (EBR) 
(2003);  and  Signori  et  al.  (2004).  Major  concepts  of  NCO  are  described  in  Alberts,  Garstka, 
and  Stein  (1999)  and  Alberts  et  al.,  (2001). 
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an  improvement  in  the  quality  of  information  will  improve  the  qual¬ 
ity  of  situational  awareness. 

OFT  and  OASD  (Nil)  tasked  RAND  to  undertake  a  study  to 
apply  the  NCO  CF  to  the  air-to-air  combat  mission.  The  primary 
objective  of  this  case  study  is  to  understand  whether  the  relationships 
between  NCO  capabilities  hypothesized  to  exist  in  the  NCO  CF  are 
valid  for  this  particular  military  mission  area  and  to  determine  how 
NCO  capability  improvements  may  lead  to  increases  in  military  force 
effectiveness  in  air-to-air  combat.  In  other  words,  our  purpose  was  to 
“learn  by  doing”  by  applying  the  NCO  conceptual  framework  to  a 
specific  mission. 

Figure  S.1 

The  NCO  Conceptual  Framework  and  Its  Application  to  Air-to-Air  Combat 


RAND  MG268-S.1 


xviii  NCO  Case  Study:  Air-to-Air  Combat  With  and  Without  Link  16 


The  air-to-air  combat  mission  was  chosen  as  an  initial  case  study 
because  it  can  involve  relatively  simple  tactical  engagement  situations 
with  a  small  number  of  aircraft.  We  anticipated  it  would  be  relatively 
easy  to  apply  the  NCO  CF  to  simple  tactical  engagements.  We  were 
also  fortunate  that  a  source  of  quantitative  force  effectiveness  data  was 
available  for  this  mission  area.  We  were  able  to  utilize  data  from  the 
Joint  Tactical  Information  Distribution  System  (JTIDS)  Operational 
Special  Project  (Hq.  USAF,  1997).  Of  particular  interest  is  that  the 
JTIDS  project  found  that  fighter  aircraft  in  air-to-air  engagements 
were  significantly  more  effective  when  equipped  with  the  Link  16 
datalink  than  when  equipped  solely  with  voice  communications.  Spe¬ 
cifically,  Link  16-equipped  fighters  saw  approximately  a  two-and-a- 
half  times  improvement  in  the  kill  ratio  (Red  aircraft  to  Blue  aircraft 
shot  down),  both  during  daylight  and  nighttime  conditions.  This 
report  examines  whether  the  NCO  CF  can  explain  this  major 
increase  in  mission  effectiveness. 

Figure  S.l  showed  the  NCO  CF  top-level  concepts  that  serve  as 
the  foundation  for  this  case  study,  with  the  numbers  indicating  the 
order  in  which  they  are  addressed  in  this  report.  The  components  of 
individual  sense-making  are  shaded  separately:  we  incorporate  the 
quality  of  individual  awareness  and  quality  of  decisions  concepts  but 
not  the  quality  of  individual  understanding  concept.  Other  concepts 
we  focused  on  relate  to  physical  NCO  capabilities-notably  the  degree 
of  networking  and  the  resulting  quality  of  information  the  pilots 
obtained  from  their  organic  sensors  as  well  as  from  the  network  to 
which  they  are  connected  (see  the  various  information  concepts  listed 
in  Figure  S.l).  On  the  other  hand,  we  did  not  employ  the  quality  of 
interactions  or  sense-making  concepts  because  we  lacked  data  for 
these  measures  at  the  time  the  study  was  conducted.  It  is  important  to 
note  that  these  concepts  represent  activities  that  take  place  in  the 
cognitive  domain  (i.e.,  mental  processes)  and  the  social  domain  (i.e., 
interactions,  such  as  conversations,  between  warfighters)  and  are  dif¬ 
ficult  to  evaluate  directly. 
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Mission  Capability  Packages  and  Networking. 

In  the  live  flight  operational  exercises  examined  as  part  of  the  JTIDS 
Operational  Special  Project,  F-15  fighter  packages  of  two  to  eight 
aircraft  flew  against  equal  size  or  larger  packages  of  enemy  aircraft. 
We  shall  designate  these  fighter  aircraft  packages  mission  capability 
packages  (MCPs). 

Two  alternative  Blue  MCPs  flew  against  the  same  Red  MCPs  in 
these  live  flight  exercises.  The  Blue  MCPs  had  different  networking 
capabilities  but  were  otherwise  identical.  One  Blue  MCP  was 
equipped  with  the  Link  16  digital  data  communications  network, 
while  the  other  had  only  voice  communications.  The  Red  fighters 
that  participated  in  these  live  flight  operational  exercises  had  voice 
only  communications. 

Voice  Networks 

Figure  S.2  illustrates  the  voice  channel  structure  typically  employed 
by  Air  Force  fighters,  in  this  case  for  an  MCP  with  four  fighters. 
Pilots  each  monitor  two  separate  voice  channels.  AWACS  broadcasts 
aircraft  track  information  on  Channel  1  to  the  Blue  aircraft.  The  four 
fighter  aircraft  communicate  among  themselves  on  Channel  2.  Each 
fighter  pilot  listens  to  two  channels  at  a  time,  and  only  one  aircraft 
pilot  can  speak  on  a  channel  at  a  time. 

Air  Force  pilots  have  developed  a  voice  coding  scheme  that 
allows  pilots  and  AWACS  flight  controllers  to  transmit  approximately 
one  aircraft  track  about  three  times  every  ten  seconds.  We  use  ten 
seconds  as  the  air  picture  track  update  cycle  time  because  this  is  the 
rate  at  which  the  AWACS  radar  antenna  rotates  or  performs  one 
complete  surveillance  cycle  of  the  battlespace.  So,  in  principle,  F-15 
fighter  pilots  can  receive  updated  air  track  information  from  AWACS 
every  ten  seconds,  if  AWACS  flight  controllers  have  the  time  to  ver¬ 
bally  transmit  this  information  over  the  voice  network  every  radar 
sweep  and  if  there  is  “time  available”  to  transmit  the  information  over 
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Figure  S.2 

Typical  Voice  Channel  Connectivity 


Channel  2:  intraflight  communications 
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the  voice  channel  (i.e. ,  if  there  is  no  contention  for  the  voice  chan¬ 
nel). 

The  pilots  must  interpret  the  spoken  information  they  receive 
on  their  voice  radios  and  build  a  mental  three-dimensional  “picture” 
of  the  positions  and  velocities  of  reported  aircraft.  This  is  known  as 
developing  situational  awareness  of  the  battlespace  and  is  a  persistent 
activity  because  of  the  dynamic  nature  of  air  combat.  While  fighter 
pilots  generally  have  the  mental  ability  to  keep  air  track  information 
in  their  minds  for  long  periods,  the  utility  of  this  information 
decreases  as  an  air  track  “age”  grows.  An  air  track  with  an  age  of  ten 
seconds  or  more  has  little  utility  because  the  pilot  will  have  only  a 
vague  idea  where  the  fast-moving  jet  fighter  may  be  (the  object  that 
corresponds  to  the  air  track),  especially  at  close  ranges.  We  approxi¬ 
mate  the  process  of  removing  old  information  from  a  fighter  pilot’s 
mental  map  or  “common  operational  picture”  of  the  battlespace  in 
the  following  way:  air  tracks  older  than  the  AWACS  update  rate  are 
“dropped”  on  the  grounds  that  the  tracked  plane  will  have  moved  far 
enough  in  ten  seconds  to  make  the  pilot’s  mental  air  track  position 
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and  velocity  estimate  too  inaccurate  to  be  useful  in  a  high-speed  tacti¬ 
cal  air  combat  engagement. 

Link  16  Data  Communications  Network 

Link  16  is  a  wireless  data  communications  system  that  provides  air 
track  and  other  information  to  fighter  aircraft,  other  weapons  plat¬ 
forms,  and  command  and  control  (C2)  nodes  equipped  with  JTIDS 
and  Multifunctional  Information  Distribution  System  (MIDS)  com¬ 
munications  terminals.  Link  16  uses  a  time-division  multiple  access 
(TDMA)  wireless  network  structure  and  a  jam-resistant,  frequency- 
hopping  waveform.  This  networking  structure  is  illustrated  in  Figure 
S.3.  In  this  type  of  network,  each  participant  (or  network  node)  can 
receive  all  transmissions  made  by  other  network  participants. 

Fighters  equipped  with  Link  16  can  receive  air  track  informa¬ 
tion  from  other  neighboring  fighter  aircraft  and  from  AWACS  (if 
AWACS  is  within  a  line  of  sight  of  the  fighter).  A  Link  16  network  is 
composed  of  128  time  slots  per  second,  with  each  slot  capable  of 

Figure  S.3 

Link  16  Network  Connectivity 
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describing  a  single  airplane  track  to  a  high  degree  of  accuracy.  Link 
16  air  tracks  received  by  a  particular  fighter  from  other  aircraft  are 
shown  on  a  display  screen  in  the  cockpit  along  with  air  tracks 
detected  by  the  aircraft’s  organic  sensors.  Therefore,  each  fighter  pilot 
in  a  Link  16-equipped  MCP  can  display  nearly  the  same  air  track 
information  or  the  same  picture  of  the  battlespace. 


Information  "Share-Ability"  and  Quality  of  Information 

The  “degree  of  information  share-ability”  concept  describes  how  well 
individual  pieces  of  information  can  be  shared  through  use  of  the 
MCP’s  networking  capabilities.  In  comparison  to  voice-only  com¬ 
munications,  the  Link  16  network  acts  as  an  information  multiplier; 
what  is  detected  by  one  aircraft  (either  by  AWACS  or  by  a  fighter)  is 
immediately  shared  with  all  other  fighters  in  the  MCP  precisely  and 
in  near  real  time.  In  contrast,  voice  transmissions  are  relatively  slow 
(maximum  of  three  track  updates  every  ten  seconds  across  an  MCP 
voice  channel),  meaning  that  only  a  small  fraction  of  the  detected 
information  can  be  shared.  Further,  voice  communications  introduce 
errors,  either  in  the  verbal  communications  themselves  or  because  of 
radio  noise  or  interference. 

Consider  the  early  stages  of  an  air-to-air  combat  engagement 
shown  in  Figure  S.4.  In  the  tactical  engagement,  four  Blue  fighters 
engage  four  Red  fighters.  The  four  Blue  fighters  are  provided  threat 
warning  information  by  AWACS  and  may  be  vectored  by  AWACS  to 
engage  particular  threat  aircraft.  The  figure  illustrates  the  “opening 
gambit,”  or  early  stages  of  the  engagement,  which  is  a  key  part  of  the 
engagement  recognized  as  strongly  influencing  the  final  outcome,  as 
the  Blue  aircraft  have  an  opportunity  to  maneuver  for  highly  advan¬ 
tageous  positions  prior  to  engaging  the  Red  aircraft  directly.  Fiere, 
the  AWACS  aircraft  has  radar  coverage  of  the  entire  battlespace.  Two 
of  the  Blue  fighters  (Blue  1 1  and  Blue  12)  have  radar  locks  on  two  of 
the  Red  fighters  (Red  1  and  2).  Two  of  the  Red  aircraft  (Red  3  and  4) 
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are  out  of  radar  range  for  any  of  the  Blue  fighters  and  are  on  the  very 
edge  of  the  battlespace  but  are  approaching  their  attack  positions 
rapidly  and  are  detected  by  AWACS. 

We  calculated  the  quality  of  information  across  the  MCP  for  the 
engagement  geometry  shown  in  Figure  S.4,  for  both  an  MCP  with 
Link  16  and  an  MCP  with  voice-only  communications.  Note  the  use 
of  the  term  “information”  to  indicate  that  it  includes  information 
available  to  the  pilot  of  a  particular  aircraft  from  both  organic  sensors 
and  the  network.  Figure  S.5  compares  quality  of  information  scores 
for  each  Blue  aircraft  in  both  the  voice-only  and  Link  16-equipped 
MCPs  along  four  metrics  (all  normalized  between  zero  and  one): 

•  Completeness  (Detection)  is  the  percentage  of  all  air  tracks  in 
the  engagement  (four  Red  and  five  Blue  aircraft)  that  the  Blue 


Figure  S.4 

Early  Stages  of  a  Tactical  Engagement 
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aircraft  either  detects  directly  or  has  reported  to  it  in  the  last  ten 
seconds; 

•  Correctness  (Identification)  is  the  percentage  of  the  air  tracks  for 
which  the  Blue  aircraft  has  correct  combat  identification  (ID) — 
i.e.,  Red,  Blue,  or  neutral/civilian  aircraft.  If  the  air  track  ID  is 
correct,  a  score  of  1 .0  is  given.  If  it  is  incorrect  or  designated  as 
unknown,  a  score  of  zero  is  given. 

•  Correctness  (Location)  is  the  percentage  of  air  tracks  for  which 
the  Blue  aircraft  has  a  location  report  (either  from  direct  detec¬ 
tion  or  network  communications  links).  If  the  location  report  is 
less  than  one  second  old,  it  is  considered  to  be  “near  real  time,” 
allowing  for  precise  maneuvering  and  cuing  fire  control  systems, 
and  has  a  value  of  1.0.  If  the  report  is  between  one  and  ten 
seconds  old,  it  is  considered  to  be  “non-real  time,”  suitable  only 
for  general  cuing,  and  has  a  value  of  0.25.  If  the  report  is  older 
than  10  seconds,  it  is  give  a  value  of  0,  as  described  earlier. 

•  Correctness  (Velocity):  Velocity  is  the  percentage  of  air  tracks 
for  which  the  aircraft  has  a  velocity  report.  As  with  Correctness 
(Location),  the  velocity  report  has  value  1.0  if  it  is  less  than  one 
second  old,  0.25  if  it  is  between  one  and  ten  seconds  old,  and  0 
if  it  is  older  than  that. 

As  shown,  Blue  aircraft  in  the  Link  16  equipped  MCP  had 
much  higher  quality  of  information  scores  than  the  voice-only  MCP, 
especially  for  the  Location  and  Velocity  metrics  (which  rely  heavily 
on  precise,  real-time  air  track  updates). 


Shared  Awareness  and  Decisionmaking 

Interviews  with  experienced  pilots  revealed  that  the  improved  quality 
of  information  under  Link  16  improved  situational  awareness  and 
subsequent  decisionmaking  in  two  ways.  First,  in  general,  the  pilots 
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Figure  S.5 

Comparing  Quality  of  Shared  Individual  Information  Across  MCPs 
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with  access  to  the  Link  16  network  reported  spending  less  time 
building  situational  awareness  (i.e.,  determining  where  the  Red  and 
Blue  aircraft  are)  than  pilots  with  access  only  to  the  voice-only  net¬ 
work.  In  the  voice-only  network,  pilots  had  to  continually  listen  to 
voice  traffic  describing  air  tracks,  mentally  convert  each  description 
into  a  velocity  and  location,  predict  where  the  aircraft  would  likely  be 
over  time  based  on  the  last  report,  and  perform  these  mental  calcula¬ 
tions  while  listening  to  further  incoming  reports.  Formal  interviews 
were  conducted  with  two  pilots  who  had  experience  with  Link 
16-equipped  aircraft.  One  of  these  pilots  was  a  key  participant  in  the 
JTIDS  operational  special  project.  In  addition,  we  have  discussed  the 
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findings  of  this  report  with  four  other  pilots  who  have  had  experience 
with  Link  16-equipped  aircraft  and  with  the  new  tactics  Link  16 
enables.  This  process  of  gaining  awareness  was  described  as  slow 
(restricted  by  voice  transmissions),  mentally  taxing,  and  potentially 
error-prone.  Further,  because  the  manual  mental  process  of  building 
awareness  is  error-prone  under  the  stressing  conditions  of  combat, 
pilot  situational  awareness  information  likely  will  not  be  entirely 
common  across  the  MCP.  In  other  words,  situational  awareness  is 
shared  and  interpreted  imperfectly  among  pilots  over  voice  channels. 

In  contrast,  in  the  Link  16  network,  pilots  are  presented  with  a 
continually  updated  image  visually  displaying  the  precise  positions 
and  velocities  of  all  detected  aircraft  in  the  battlespace.  The  resulting 
process  of  gaining  situational  awareness  was  much  faster,  almost 
automatic  (no  mental  calculations  required),  and  accurate.  The 
resulting  time  compression  in  obtaining  information  and  awareness 
with  Link  16  is  shown  in  Figure  S.6.  This  freed  time  could  be  used  to 
consider  more  alternative  courses  of  action,  which  will  tend  to  lead  to 
better  decisions,  and  make  more  decisions  in  a  given  period  of  time, 
which  (assuming  the  decisions  are  reasonable)  should  lead  to  more 
targets  destroyed.  Notably,  the  freed  time  also  allows  the  wingman 
time  for  sense-making  and  making  decisions  to  engage  targets,  as 
opposed  to  spending  virtually  all  their  time  gathering  and  monitoring 
critical  information  as  in  conventional  doctrine. 

Second,  the  pilots  were  able  to  improve  execution  of  air  combat 
tactics  that  were  enabled  by  taking  advantage  of  their  increased 
awareness  as  well  as  the  increased  time  they  had  available  for  decision¬ 
making.  From  the  interviews,  we  have  identified  four  broad  types  of 
improved  tactics  for  air-to-air  combat.  These  tactics  are  illustrated  in 
Figure  S.7. 

The  first  of  these  is  simply  an  increased  number  of  engagements 
in  the  same  period.  This  tactic  is  possible  because  pilots  with  Link  16 
can  quickly  recognize  the  most  efficient  attack  trajectories.  This  is  an 
important  consideration  because  (according  to  the  pilots)  the  fighters 


Summary  xxvii 


Figure  S.6 

Decision  Speed  and  Competitive  Advantage  with  Link  16 
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only  have  a  limited  time  to  engage  before  they  run  out  of  fuel  and 
must  return  to  base. 

The  second  is  the  employment  of  the  wingman  as  a  combatant 
rather  than  as  a  defensive  patroller.  With  Link  16  and  good  combat 
ID  capabilities,  the  location  and  identity  of  threat  aircraft  are  appar¬ 
ent  to  pilots  of  all  Blue  aircraft.  The  flight  lead  has  more  options  for 
employing  the  wingman  as  a  primary  shooter  because  of  the  higher 
levels  of  individual  and  shared  understanding  of  the  engagement, 
effectively  doubling  the  firepower.  Wingmen  do  not  have  to  take  up 
defensive  positions  to  hedge  against  possible  attacks  from  threat  air¬ 
craft  advancing  from  unknown  locations. 

The  third  is  the  use  of  other  planes’  track  information  to  vector 
earlier  and  more  accurately,  which  allows  a  Blue  fighter  to  enter  an 
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Figure  S.7 

Improved  Air-to-Air  Tactics  Execution  Enabled  by  Improved  Awareness 


1.  Increased  numbers  of  engagements 
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2.  Employment  of  the  wingman  as 
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patroller 
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3.  Earlier  and  more  accurate  vectoring 
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formations  to  trap  and 
destroy  Red  aircraft 


SOURCE:  Interviews  with  fighter  pilots  experienced  with  Link  16. 
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engagement  from  a  position  of  maximum  advantage,  before  the  Blue 
fighter’s  radar  (or  the  Red  plane’s  radar,  for  that  matter)  can  detect 
the  engaging  plane.  This  tactic  takes  maximum  advantage  of  AWACS 
or  other  offboard  sensor  threat-reporting. 

The  fourth  is  the  use  of  “ambush”  combat  air  patrols  (CAPs) 
and  the  use  of  terrain  to  trap  and  destroy  Red  aircraft.  Because  all 
Blue  aircraft  locations  are  known  by  all  Blue  fighter  pilots — even  if 
those  aircraft  are  operating  in  voice  communications  or  Identifica¬ 
tion,  Friend  or  Foe  (IFF),  transponder  silence — they  have  more 
options  to  engage  targets.  One  example  of  an  “ambush  CAP”  tactic  is 
when  a  Blue  fighter  chases  a  Red  fighter  towards  other  Blue  aircraft. 
The  latter  Blue  aircraft  have  their  radars  turned  off  (or  are  hiding  in  a 
canyon)  so  the  Red  fighter  is  not  likely  to  know  that  the  latter  Blue 
aircraft  are  present.  Then,  when  the  Red  aircraft  is  chased  into  range, 
the  other  Blue  fighters  will  suddenly  engage  the  Red  fighter,  surpris- 
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ing  the  Red  pilot  and  likely  destroying  the  Red  fighter  with  minimal 
risk  to  Blue  aircraft.  This  latter  tactic  is  an  example  of  tactical  self¬ 
synchronization  enabled  by  the  Link  16  network. 


Mission  Effectiveness 

Loss  exchange  ratios  (number  of  Red  aircraft  killed  divided  by  the 
number  of  Blue  aircraft  killed)  from  the  JTIDS  Operational  Special 
Project  are  shown  in  Table  S.l.  It  is  based  on  the  results  of  12,000 
training  sorties  in  tactical  air-to-air  combat.  On  average,  Link  16  led 
to  a  two-and-half  times  improvement  in  the  kill  ratio  (Red  aircraft  to 
Blue  aircraft  shot  down),  during  both  daylight  and  nighttime  condi¬ 
tions. 

The  analysis  presented  above  indicates  in  these  tactical  engage¬ 
ments  the  better  decisionmaking  and  improved  tactics  execution  by 
Blue  fighter  pilots  were  enabled  by  improved  situational  awareness 
provided  by  Link  16.  This  in  turn  led  to  the  Link  16-equipped 
MCPs’  improvements  in  kill  ratios.  This  chain  of  inferences  is  veri¬ 
fied  by  our  interviews  with  experienced  pilots.  For  some  steps  of  the 
NCO  inference  chain,  quantitative  data  were  not  available — e.g.,  data 
monitoring  how  pilots  gained  awareness  and  made  decisions  during 
the  engagements.  Nevertheless,  pilot  interviews  substantiate  the 
validity  of  the  inference  chain  described  in  the  NCO  framework  for 
this  mission  area. 


Table  S.l 

Results  of  the  JTIDS  Operational  Special  Project 
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Conclusions 

Figure  S.8  presents  averages  of  the  NCO  CF  metric  scores  we  calcu¬ 
lated  in  this  case  study  across  the  MCPs.  As  shown,  despite  starting 
with  similar  airframes,  training,  doctrine,  and  organic  sensing  capa¬ 
bilities,  the  Link  1 6-equipped  MCP  was  able  to  take  advantage  of  the 
information  shared  within  the  MCP  through  Link  16  and  voice  net¬ 
works  far  more  effectively  than  the  voice-only  MCP.  As  hypothesized 
by  the  NCO  tenets,  the  robustly  networked  force  enabled  via  Link  16 
improved  information  sharing  and  the  resulting  quality  of  informa¬ 
tion,  which  enhanced  shared  situational  awareness,  which  in  turn 


Figure  S.8 

Summary  Comparison  of  MCPs  Using  Average  Scores 
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enabled  self-synchronization  (in  this  case  study,  as  measured  by  the 
ability  to  make  improved  decisions  and  execute  improved  tactics)  and 
which  resulted  in  dramatically  increased  mission  effectiveness  as 
measured  by  the  kill  ratios. 

We  have  applied  the  NCO  CF  and  developed  quantitative  esti¬ 
mates  for  key  NCO  metrics  within  the  framework.  We  have  exam¬ 
ined  several  inference  chains  that  run  through  the  NCO  CF  and  have 
found  them  to  be  consistent  with  the  results  of  a  key  air-to-air  live 
flight  experiment  and  with  the  observations  of  experienced  combat 
pilots. 

Finally,  we  recommend  additional  case  studies  be  performed  of 
more  complex  mission  areas  and  that  extend  this  analysis  of  the  air- 
to-air  mission  area  further  to  provide  further  understanding  of  NCW 
and  the  NCO  CF,  and  particularly  of  the  cognitive  and  social 
domain  concepts,  attributes  and  metrics  covered  in  the  framework 
(e.g.,  cognitive  measures  of  sense-making  and  interactions). 
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Introduction 


Overview 

The  Joint  Tactical  Information  Distribution  System  (JTIDS)  Opera¬ 
tional  Special  Project  found  that  fighter  aircraft  in  air-air  engage¬ 
ments  were  significantly  more  effective  when  equipped  with  the  Link 
16  datalink  than  when  only  equipped  with  voice  communications 
(Hq.  USAF,  1997).  The  project  has  been  cited  as  “compelling  evi¬ 
dence”  (Alberts  et  ah,  2001,  p.  2 44)  in  favor  of  the  Network-Centric 
Warfare  (NCW)  “central  hypothesis”  that  a  robustly  networked  force 
(in  this  case,  aircraft  with  Link  16),  possessing  capabilities  and  attri¬ 
butes  fully  exploiting  its  networking  capabilities,  will  be  able  to  gen¬ 
erate  increased  combat  power  (Alberts  et  ah,  2001,  pp.  57-58). 

However,  there  was  only  limited  understanding  of  how  and  why 
these  improvements  arose.  The  Office  of  Force  Transformation 
(OFT)  and  the  Office  of  the  Assistant  Secretary  of  Defense  for  Net¬ 
works  and  Information  Integration  (OASD  [Nil])  sponsored  RAND 
to  undertake  a  study  using  the  Network-Centric  Operations  Concep¬ 
tual  Framework  (NCO  CF)1  to  understand  why  these  improvements 
occurred.  A  key  purpose  of  the  case  study  was  to  determine  whether 
the  explanation  for  these  improvements  provided  support  for  the  ten¬ 
ets  of  NCW: 


1  The  NCO  CF  is  described  in  Signori  et  al.  (2002);  EBR  (2003);  and  Signori  et  al.  (2004). 
Major  concepts  of  NCO  are  described  in  Alberts,  Garstka,  and  Stein  (1999)  and  Alberts  and 
Garstka  (2001). 
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•  “A  robustly  networked  force  improves  information  sharing 

•  “Information  sharing  enhances  the  quality  of  information  and 
shared  situational  awareness 

•  “Shared  situational  awareness  enables  self-synchronization,  and 
enhances  sustainability  and  speed  of  command 

•  “These,  in  turn,  dramatically  increase  mission  effectiveness.” 
(Alberts  and  Garstka,  2001.) 

The  NCW  tenets  form  a  chain  of  hypotheses  leading  from 
improvements  in  networking  capabilities  to  eventual  mission  effec¬ 
tiveness.  The  NCO  CF  expands  and  formalizes  these  hypotheses.  It 
provides  a  detailed  framework  of  NCW-related  force  attributes  (such 
as  networking,  information  sharing,  and  situational  awareness)  and 
hypotheses  for  how  these  attributes  influence  each  other.  The  result  is 
a  detailed  and  interlinked  set  of  hypotheses  for  how  networking 
improvements — and  procedures  taking  advantage  of  the  networking 
improvements — might  lead  to  improved  combat  effectiveness. 
Importantly,  the  NCO  CF  describes  metrics  for  assessing  a  force’s 
possession  of  these  attributes,  making  it  possible  to  test  whether  and 
how  attribute  possession  relates  to  improved  combat  effectiveness. 

In  this  report,  the  NCO  CF  is  applied  to  an  example  air-to-air 
combat  mission,  typical  of  those  observed  in  the  JTIDS  study.  The 
resulting  analysis  compares  assigning  two  air  combat  mission  capabil¬ 
ity  packages  (MCPs)  to  the  example  mission,  identical  in  every  way 
except  for  one  of  them  being  equipped  with  Link  16  and  the  other 
equipped  with  traditional  radio  communications.  While  not  fully 
conclusive  because  of  lack  of  data,  the  main  conclusions  of  the  analy¬ 
sis  were  that  the  observed  improvements  in  combat  effectiveness  like¬ 
ly  were  consistent  with  both  the  NCW  tenets  and  the  more  detailed 
hypotheses  of  NCO  CF.  In  brief,  Link  16’s  networking  capabilities 
dramatically  improved  the  quality  and  distribution  of  information  to 
the  pilots  in  the  example  mission,  resulting  in  an  increase  to  both 
individual  and  shared  information.  This  is  believed  to  be  associated 
with  an  increase  in  awareness,  which  resulted  in  the  ability  to  employ 
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advanced  tactics  depending  on  enhanced  awareness.  Finally,  the  use 
of  advanced  tactics  is  believed  to  have  led  to  the  improved  effective¬ 
ness  observed  in  the  JTIDS  Special  Operations  Project. 


The  NCO  Conceptual  Framework 

At  the  top  level,  the  NCO  CF  comprises  a  set  of  concepts  corre¬ 
sponding  to  the  major  NCW-related  capabilities  of  interest  and 
hypothesized  dependencies  between  them.2  Each  dependency  is  a 
hypothesis  of  the  form  that  “improvement  for  one  concept  leads  to 
improvements  in  its  dependent  concepts.”  Figure  1.1  shows  the  top 
level  of  the  NCO  CF. 

The  top-level  concepts  and  their  hypothesized  dependencies  are 
described  below. 

Force  comprises  the  elements  of  a  military  force,  including  the 
resources  and  processes  assigned,  utilized,  or  deployed  in  support  of  a 
mission.  These  include  information  sources  (such  as  sensors),  value- 
added  services  that  process  and  distribute  the  information,  command 
and  control  (C2)  elements,  and  effectors — the  warfighters,  weapons, 
and  other  systems  that  physically  can  destroy  adversaries  or  affect 
other  elements  in  the  battlespace.  The  force  collectively  generates  a 
set  of  organic  information  and  a  networking  capability  for  processing 
and  sharing  that  information. 

Quality  of  organic  information  assesses  the  quality  of  information 
at  its  source  (i.e.,  when  it  is  first  collected).  Here,  “quality”  results 
from  several  contributing  factors,  including  the  completeness,  cor¬ 
rectness,  timeliness,  and  relevance  of  the  information. 

Degree  of  networking  measures  the  extent  to  which  force  entities 
are  interconnected  and  the  quality  (speed,  accuracy,  reliability,  and 


2  The  NCO  concepts  found  in  the  NCO  CF  allow  us  to  measure  the  extent  to  which  a 
given  concept  is  present  or  is  instantiated  in  the  force. 
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Figure  1.1 

Top  Level  of  the  NCO  Conceptual  Framework 
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assurance)  of  those  connections  in  exchanging  data  under  a  variety  of 
possible  scenarios. 

Degree  of  information  “share-ability  ”  applies  the  degree  of  net¬ 
working  to  the  force’s  organic  information.  It  measures  the  degree 
and  quality  to  which  force  entities  can  share  the  information  in  their 
possession  with  other  force  entities. 

Quality  of  interactions  assesses  the  degree  to  which  force  mem¬ 
bers  interact  with  each  other  and  the  quality  of  those  interactions. 
This  concept  applies  to  all  interactions  between  force  members,  not 
just  technical  network-supported  data  exchanges.  It  is  concerned  with 
interactions  between  force  members  at  multiple  levels,  ranging  from 
basic  information  sharing  to  detailed  characterizations  of  how 
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effectively  an  organization’s  members  work  together  to  accomplish 
mission  objectives.  Consequently,  this  concept  has  many  influences 
throughout  the  NCO  CF,  starting  with  contributing  to  the  quality  of 
information  an  individual  receives  from  across  the  force. 

Qiiality  of  individual  information  assesses  the  quality  of  the 
information  each  individual  in  the  force  has  in  his  or  her  possession 
from  all  sources,  whether  generated  organically,  transmitted  over  the 
technical  network,  or  heard  in  a  conversation. 

Degree  of  shared  information  assesses  the  quality  of  the  informa¬ 
tion  held  in  common  by  groups  of  force  members.  In  addition  to 
assessing  quality,  this  concept  assesses  the  extent  to  which  the  infor¬ 
mation  is  shared  across  the  group  and  to  what  degree  of  consistency. 

The  sense-making  concepts  jointly  describe  how  well  an  individ¬ 
ual  or  group  “makes  sense”  of  the  information  in  their  possession, 
and  determines  what  to  do  in  response. 

Qiiality  of  individual  awareness  describes  how  well  an  individual 
can  interpret  the  information  in  their  possession  into  a  mental  view 
of  the  battlespace  that  includes  mission  constraints,  environmental 
factors,  time-space  relationships,  and  the  capabilities  and  intentions 
of  Red,  Blue,  and  neutral  forces,  along  with  attendant  uncertainties. 

Qiiality  of  individual  understanding  describes  how  effectively  the 
individual  can  infer  meaning  from  their  mental  view  of  the  battle- 
space,  including  recognition  of  patterns,  cause-effect  relationships, 
dynamic  futures,  and  opportunities  and  risks. 

Qiiality  of  individual  decisions  measures  how  well  an  individual’s 
choices  build  on  his  or  her  awareness  and  understanding  and  how 
appropriate  those  choices  are  for  the  situation. 

The  shared  sense-making  concepts  are  similar  to  the  individual 
sense-making  concepts.  Like  the  shared  information  concept,  they 
address  the  degree  and  consistency  of  awareness,  understanding,  and 
decisions  shared  across  the  group,  in  addition  to  quality. 

Degree  of  decision/plan  synchronization  assesses  whether  the  com¬ 
plete  set  of  decisions  made  by  the  force  are  synchronized  with  each 
other  (i.e.,  mutually  reinforcing)  or  at  least  deconflicted. 
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Degree  of  actions! entities  synchronization  assesses  the  execution  of 
the  decisions.  It  examines  the  actions  of  the  force,  determining 
whether  those  actions  are  synchronized. 

Finally,  degree  of  effectiveness  assesses  the  force’s  actions  by  evalu¬ 
ating  the  force’s  achievement  of  mission  objectives  and  avoidance  of 
costs.  Of  additional  interest  is  the  force’s  degree  of  agility,  which 
measures  the  force’s  ability  to  operate  effectively  and  efficiently  in  an 
uncertain  environment. 

The  NCO  CF  associates  each  top-level  concept  with  subsidiary 
attributes.  For  example,  Chapter  Two  notes  that  the  case  study  analy¬ 
sis  focuses  on  assessing  the  quality  of  air  tracks  known  to  the  fighter 
pilots.  The  NCO  CF  assesses  the  quality  of  information  elements 
(whether  organic,  individual,  or  shared)  through  eight  subsidiary 
attributes,  shown  in  Table  1.1.  As  shown,  the  attributes  are  divided 
into  objective  attributes  (which  are  independent  of  mission  context) 
and  fitness  for  use  attributes  (which  are  context-dependent) . 

Table  1.1 

Subsidiary  Attributes  for  Quality  of  Information 


Subsidiary  Attribute 

Definition 

Objective  Attributes 

Measure  quality  in  reference  to  situation-independent 
criteria 

Correctness 

Extent  to  which  information  is  consistent  with  ground 
truth 

Consistency 

Extent  to  which  information  is  consistent  with  prior 
information 

Currency 

Age  of  information 

Precision 

Level  of  measurement  detail  of  information  item 

Fitness  for  Use  Attributes 

Measure  quality  in  reference  to  situation-dependent 
criteria 

Completeness 

Extent  to  which  information  relevant  to  ground  truth  is 
collected 

Accuracy 

Appropriateness  of  precision  of  information  to  a 
particular  use 

Relevance 

Proportion  of  information  collected  that  is  related  to 
task  at  hand 

Timeliness 

Extent  to  which  currency  of  information  is  suitable  to  its 
use 
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When  applied,  a  subsidiary  attribute  is  associated  with  one  or 
more  metrics  that  quantitatively  assess  the  joint  force  with  respect  to 
the  attribute.3  The  body  of  this  report  describes  the  calculation  of  a 
representative  set  of  these  metrics  for  a  typical  air-to-air  combat  mis¬ 
sion.  The  report  also  describes  modeling  the  attributes’  dependencies 
through  multidimensional  quantitative  relationships  between  the 
attributes’  metrics. 


The  JTIDS  Operational  Special  Project  and  an  Example  Air 
Combat  Mission 

The  JTIDS  Operational  Special  Project  examined  the  results  of 
12,000  training  sorties  in  tactical  air-to-air  combat  (the  sorties  were 
carried  out  as  part  of  the  regular  pilot  training  program).  Each  train¬ 
ing  sortie  featured  a  Blue  mission  capability  package  (MCP),  includ¬ 
ing  an  Airborne  Warning  and  Control  System  (AWACS)  aircraft  plus 
fighter  aircraft.  The  size  of  the  engagements  ranged  from  two  Blue 
fighters  on  two  Red  fighters  to  eight  Blue  fighters  on  1 6  Red  fighters. 
Engagements  occurred  during  the  day  and  at  night.  Within  the 
JTIDS  project,  the  primary  independent  variable  was  whether  or  not 
the  Blue  MCP  was  equipped  with  the  Link  16  data  network.  The 
capability  of  the  Red  aircraft  remained  consistent  during  the  project. 
The  JTIDS  Operational  Special  Project  provided  the  kill  ratios  seen 
in  Table  1.2.  It  is  based  on  the  results  of  12,000  training  sorties  in 
tactical  air-to-air  combat.  On  average,  Link  16  led  to  a  two-and-half 
times  improvement  in  the  kill  ratio  (Red  aircraft  to  Blue  aircraft  shot 
down),  during  both  the  daylight  and  nighttime  conditions. 

The  analysis  presented  above  indicates  that  in  these  tactical 
engagements  the  better  decisionmaking  and  use  of  advanced  tactics 


3  The  NCO  CF  contains  literally  hundreds  of  subsidiary  attributes  and  metrics,  so  they  are 
not  further  identified  here.  This  report  lists  a  significant  percentage  of  them  in  the  process  of 
applying  the  NCO  CF’s  measures  and  metrics  to  an  air  combat  mission. 
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Table  1.2 

Results  of  the  JTIDS  Operational  Special  Project 


Kill  Ratio 

Voice  Only 

Voice  Plus  Link  16 

(MCP  1) 

(MCP  2) 

Day 

3.10:1 

8.11:1 

Night 

3.62:1 

9.40:1 

employed  by  Blue  fighter  pilots  were  enabled  by  Link  16-driven 
improvements  to  situational  awareness.  This  in  turn  led  to  the  Link 
16-equipped  MCPs’  improvements  in  kill  ratios.  This  chain  of  infer¬ 
ences  is  verified  qualitatively  by  our  interviews  with  experienced 
pilots.  For  some  steps  of  the  NCO  inference  chain,  quantitative  data 
were  not  available — e.g.,  data  monitoring  how  pilots  gained  aware¬ 
ness  and  made  decisions  during  the  engagements.  Nevertheless,  pilot 
interviews  substantiate  the  validity  of  the  inference  chain  described  in 
the  NCO  framework  for  this  mission  area. 

To  provide  insight  into  what  might  have  occurred  during  the 
sorties,  Figure  1.2  shows  a  typical  combat  mission.  It  is  a  four-on- 
four  combat  scenario,  in  which  four  Blue  fighters  battle  four  Red 
fighters,  with  the  four  Blue  fighters  directed  by  an  AWACS  aircraft. 

Consider  the  early  stages  of  an  air-to-air  combat  engagement 
shown  in  Figure  1.2.  In  the  tactical  engagement,  four  Blue  fighters 
engage  four  Red  fighters.  The  four  Blue  fighters  are  provided  threat¬ 
warning  information  by  AWACS  and  may  be  vectored  by  AWACS  to 
engage  particular  threat  aircraft.  The  figure  illustrates  the  “opening 
gambit,”  or  early  stages  of  the  engagement,  which  is  a  key  part  of  the 
engagement  recognized  as  strongly  influencing  the  final  outcome 
because  the  Blue  aircraft  have  an  opportunity  to  maneuver  for  highly 
advantageous  positions  prior  to  engaging  the  Red  aircraft  directly. 
Fiere,  the  AWACS  aircraft  has  radar  coverage  of  the  entire  battle- 
space.  Two  of  the  Blue  fighters  (Blue  11  and  Blue  12)  have  radar 
locks  on  two  of  the  Red  fighters  (Red  1  and  2).  Two  of  the  Red  air¬ 
craft  (Red  3  and  4)  are  out  of  radar  range  for  any  of  the  Blue 
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Figure  1.2 

Air-to-Air  Scenario  Exemplar 
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fighters  and  are  on  the  very  edge  of  the  battlespace  but  are  approach¬ 
ing  their  attack  positions  rapidly  and  are  detected  by  AWACS. 

We  calculated  the  quality  of  shared  information  across  the  MCP 
for  the  engagement  geometry  shown  in  Figure  S.4,  for  both  an  MCP 
with  Link  16  and  an  MCP  with  voice-only  communications.  Note 
that  we  use  the  term  “shared  information”  to  indicate  that  it  includes 
the  information  available  to  the  pilot  of  a  particular  aircraft  from 
both  organic  sensors  and  from  the  network. 

To  gain  insight  into  whether  the  observed  improvements  in  air- 
to-air  combat  effectiveness  occurred  in  a  way  consistent  with  the 
NCW  hypotheses,  we  apply  the  NCO  CF  to  the  air-to-air  combat 
mission,  which  is  illustrated  in  Figure  1.2.  Restricting  the  analysis  to 
a  single  mission  has  important  implications,  both  positive  and  nega¬ 
tive.  On  the  positive  side,  it  limits  and  simplifies  the  analysis  required 
to  generate  metrics.  Thus,  we  can  direct  most  of  our  attention  to 
understanding  the  application  of  the  NCO  CF’s  metrics — important 
for  a  first  application  of  the  framework.  Using  an  illustrative  scenario 
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also  permits  use  of  the  NCO  CF  despite  not  having  a  full  set  of  scien¬ 
tifically  captured  data  about  what  happened  and  how  systems  per¬ 
formed  and  were  used  during  the  JTIDS  Operational  Special  Project. 
On  the  negative  side,  this  analysis  has  certain  gaps  because  quantita¬ 
tive  data  were  unavailable  to  calculate  certain  metrics.  The  entire  set 
of  possible  and  important  chains  of  influences  articulated  in  the 
NCO  CF  have  not  been  examined  in  this  case  study.  Furthermore, 
the  simple  model  does  not  incorporate  the  range  of  engagement  sizes 
that  took  part  during  the  experiments  or  the  ways  in  which  each  of 
those  experiments  evolved. 


Outline  of  the  Report 

Chapter  Two  of  the  paper  describes  the  overall  methodology  for 
applying  the  NCO  CF  and  calculating  the  resulting  metrics  for  the 
air-to-air  combat  mission.  The  chapter  describes  the  data  available  for 
the  study  and  the  resulting  selection  of  concepts,  subsidiary  attri¬ 
butes,  and  metrics  for  analysis.  It  also  describes  the  bases  for  calculat¬ 
ing  the  metrics,  and  the  relationships  between  the  metrics. 

Chapters  Three  to  Ten  describe  the  calculation  of  metrics  for 
each  selected  concept.  Each  of  these  chapters  first  describes  the  factors 
contributing  to  that  concept.  Each  chapter  then  describes  the  calcula¬ 
tion  of  the  corresponding  metrics,  presents  the  results,  and  concludes 
by  presenting  a  figure  providing  an  at-a-glance  comparison  between 
the  performance  of  the  Link  16— equipped  MCP  and  the  performance 
of  the  voice-only  MCP. 

Chapter  Eleven  presents  research  conclusions  and  areas  for  fur¬ 
ther  research. 

Finally,  the  appendix  describes  the  details  of  the  technical  model 
that  actually  calculated  the  measurements,  implemented  in  the  soft¬ 
ware  package  Analytica. 


CHAPTER  TWO 


Methodology 


The  methodology  uses  selected  metrics  from  the  NCO  CF,  which  is 
described  below  and  in  the  principal  references  that  are  source  docu¬ 
ments  for  this  analysis.1  The  aggregation  of  the  data,  explained  before 
the  analysis  associated  with  each  area  of  the  NCO  CF,  is  also  dis¬ 
cussed  below. 


Application  of  the  NCO  CF 

Figure  2.1  shows  the  data  available  for  assessing  each  of  the  NCO  CF 
top-level  concepts.  From  the  JTIDS  Operational  Special  Project,  we 
have  the  kill  ratios  shown  in  Table  1.1.  We  obtained  information  on 
the  technical  capabilities  of  the  voice  and  Link  16  networks  for  com¬ 
bat  aircraft  and  AWACS  from  a  variety  of  sources,  including  those 
listed  in  the  references.  In  addition  to  describing  the  characteristics  of 
the  network,  we  also  used  the  information  from  these  background 
references  and  from  interviews  with  experienced  combat  pilots  to 
construct  the  opening  phases  of  an  archetypal  four-on-four  air  com¬ 
bat  tactical  engagement.  We  interviewed  several  pilots  who  partici- 


1  The  NCO  CF  is  described  in  David  A.  Signori  et  al.  (2002);  EBR  (2003);  and  Signori  et 
al.  (2004).  Major  concepts  of  NCO  are  described  in  Alberts,  Garstka,  and  Stein  (1999)  and 
Alberts  et  al.  (2001). 
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Figure  2.1 

The  NCO  Framework  and  Corresponding  Data  Available  for  the  Case  Study 


pated  in  the  JTIDS  experiments,  which  provided  important  data  on 
the  types  of  decisions  they  were  able  to  make,  and  the  situational 
awareness  information  they  obtained  with  both  the  voice-only  and 
Link  16  network  architectures.  We  also  used  these  interviews  to  verify 
that  the  tactical  engagement  geometry  was  reasonable  and  representa¬ 
tive  of  real  tactical  engagements. 

Figure  2.2  summarizes  the  concepts  of  the  NCO  CF  addressed 
in  this  study.  The  concepts  we  incorporate  are  indicated  on  the  leg¬ 
end.  The  numbers  on  the  concepts  indicate  the  order  in  which  they 
are  addressed  in  this  report  and  are  used  to  refer  to  this  diagram  later 
in  this  report. 
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Figure  2.2 

Concepts  Included  in  the  Case  Study 
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Because  of  the  limits  of  the  data  available  from  the  JTIDS 
Operational  Special  Project,  as  well  as  the  nature  of  the  experiment 
(comparing  the  impacts  of  two  different  networking  architectures), 
this  case  study  does  not  cover  all  of  the  concepts  in  NCO  CF.  The 
subsidiary  concepts  of  individual  sense-making  are  colored  separately: 
we  incorporate  the  quality  of  individual  awareness  and  quality  of 
individual  decisions  concepts  but  not  the  quality  of  individual  under¬ 
standing  concept.  We  were  unable  to  gather  quantitative  data  to 
estimate  measurements  for  the  quality  of  interactions,  decision  syn¬ 
chronization,  and  action/entity  synchronization  concepts  in  our 
analysis.  We  were,  however,  able  to  extrapolate  shared  awareness  from 
the  data  available.  For  example,  pilot  interviews  reveal  that  Link  16 
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significantly  increases  the  level  of  individual  and  shared  situational 
awareness  available  throughout  the  entire  Blue  fighter  package.  These 
interviews  also  reveal  that  the  improvements  in  mission  effectiveness 
that  stemmed  from  increased  situational  awareness  were  a  function  of 
the  level  of  training  and  the  use  of  common  tactics,  techniques,  and 
procedures.  It  is  important  to  note  that  the  sense-making  and  deci¬ 
sion  synchronization  measures  represent  activities  that  take  place  in 
the  cognitive  domain  (i.e.,  mental  processes)  and  social  domain  (i.e., 
interactions  between  warfighters,  such  as  conversations)  and  are  diffi¬ 
cult  to  evaluate  quantitatively.  For  this  reason  and  because  of  the  lack 
of  data,  quantitative  metrics  for  shared  awareness,  shared  under¬ 
standing,  and  shared  decisionmaking  were  not  evaluated. 


Approach  to  Measurement  in  the  Case  Study 

In  the  air-to-air  example,  the  metrics  constructed  track  the  informa¬ 
tion  or  situational  awareness  positions  of  the  Blue  MCP.  Each  metric 
contains  an  array  of  subsidiary  values,  with  array  entries  tracking 
either  what  each  Blue  aircraft  pilot  “knows”  about  other  air  tracks  in 
the  battlespace  or  what  each  Blue  aircraft  pilot  is  able  to  do  as  a 
result. 

In  this  case  study,  what  each  pilot  “knows”  about  a  particular  air 
track  is  characterized  by  three  of  the  eight  quality  of  information 
attributes  discussed  in  Chapter  One — completeness,  correctness,  and 
accuracy.  These  three  attributes  most  directly  characterize  whether  a 
pilot’s  knowledge  of  air  tracks  is  sufficient  to  make  tactical  decisions 
to  take  appropriate  actions.  For  the  other  five  attributes,  we  do  not 
examine  relevance  because  all  air  tracks  in  the  engagement  battlespace 
are  assumed  to  be  relevant.  We  did  not  include  consistency  because 
we  assume  that  the  Blue  aircraft’s  sensors  reliably  report  the  tracks 
they  detect.  Accuracy  is  used  in  place  of  precision;  the  latter  simply 
describes  the  potential  margin  of  error  of  the  pilot’s  knowledge, 
whereas  accuracy  also  reflects  whether  the  margin  of  error  is  small 
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enough  to  meet  the  pilots’  needs.  As  will  be  discussed,  the  currency 
and  timeliness  of  the  information  implicitly  impact  the  calculation  of 
the  correctness  metrics  and  so  are  not  considered  separately. 

We  define  one  subsidiary  metric  to  measure  completeness,  one 
to  measure  correctness,  and  two  to  measure  accuracy  of  the  air  tracks. 

•  Completeness:  Detection,  which  tracks  whether  the  Blue  pilot  is 
aware  of  the  track  at  all.  This  metric  has  score  1  if  the  pilot  is 
aware  of  a  track  and  0  if  not. 

•  Correctness:  ID,  which  tracks  whether  the  Blue  pilot  correctly 
labels  the  air  track  as  Red  or  Blue.  This  metric  has  score  1  if  the 
pilot  correctly  labels  the  track  and  0  if  not. 

•  Accuracy:  Location,  which  tracks  the  pilot’s  awareness  of  the 
aircraft’s  location.  The  scores  of  this  metric  are  based  on  the 
pilots’  needs  for  precision  in  track  location.  Thus,  the  metric  has 
score  1  if  the  precision  is  sufficient  for  the  pilot  to  make  preci¬ 
sion  maneuver  and  targeting  decisions,  0.25  if  the  precision  is 
sufficient  to  make  general  cuing  decisions,  and  0  if  the  precision 
is  insufficient  even  for  general  cuing  decisions. 

•  Accuracy:  Velocity,  which  tracks  the  pilot’s  awareness  of  the  air 
track’s  velocity.  Like  Accuracy:  Location,  the  scores  of  this 
metric  are  based  on  the  pilots’  needs  for  accuracy  in  track 
velocity,  and  it  has  the  same  1,  0.25,  and  0  scores. 

These  four  metrics  are  the  most  important  characterizations  of  an  air 
track  for  an  Air  Force  pilot.  Note  that  air  track  currency,  measured  by 
the  latency  of  the  most  recent  track  update  reported  to  a  pilot,  affects 
the  location  and  velocity  metrics  calculations.  In  particular,  greater 
latencies  in  updates  lead  to  uncertainty  in  the  true  position  and 
velocity  of  the  tracked  aircraft,  increasing  the  potential  margin  of 
error  and  reducing  the  location  and  velocity  scores.  Chapter  Four 
presents  the  specific  formulas  incorporating  air  track  currency. 

The  case  study  uses  two  major  types  of  arrays  tracking  knowl¬ 
edge  of  air  tracks.  The  first  assesses  knowledge  of  the  raw  data  ele- 
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ments  in  the  case  study — the  17  sensed  air  tracks  in  the  battlespace 
described  in  Figure  1.2.  To  generate  the  17  tracks,  note  that  each 
Blue  aircraft  detects  itself  (five),  Blue  11  and  Blue  12  each  detect  Red 
1  and  Red  2  (four),  and  AWACS  detects  every  other  aircraft  in  the 
battlespace  with  its  radar  (eight).  This  array  type  is  used  in  assessing 
what  raw  data  each  aircraft  senses  organically  (quality  of  organic 
information)  and  how  well  the  voice-only  and  Link  16-equipped 
MCPs  share  the  raw  data  elements  across  the  Blue  aircraft  (degree  of 
information  share-ability).  Figure  2.3  shows  an  example  array.  As 
shown,  each  entry  assesses  a  pilot’s  knowledge  of  one  of  the  sensed  air 
tracks,  as  measured  by  one  of  the  four  quality  metrics. 

Figure  2.3 

An  Array  Tracking  Pilots'  Knowledge  of  Sensed  Air  Tracks 
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The  second  type  assesses  knowledge  of  the  essential  elements  of 
information  (EEIs)  in  the  case  study — the  nine  unique,  or  “fixed,”  air 
tracks  in  the  battlespace  (AWACS,  four  Blue  fighters,  and  four  Red 
fighters) .  A  Blue  aircraft  generates  its  knowledge  of  an  EEI  by  fusing 
the  multiple  sensed  tracks  corresponding  to  that  EEI;  our  analysis 
includes  a  simple  representation  of  the  fusion  process.  The  EEIs  are 
the  “refined”  elements  of  information  used  by  pilots  to  generate 
awareness  and  make  decisions,  so  this  second  array  type  is  used  to 
compute  quality  of  individual  information  and  degree  of  shared 
information  metrics.  This  second  array  type  is  used  to  compute  qual¬ 
ity  of  individual  awareness  metrics,  as  well;  we  develop  a  simple 
model  mapping  EEIs  to  “elements  of  awareness,”  which  assess  what 
the  Blue  pilots  cognitively  know  about  aircraft  tracks  in  the  battle- 
space.  Figure  2.4  shows  an  example  array.  As  shown,  it  is  identical  to 
Figure  2.3  except  for  the  change  to  fixed  tracks  rather  than  sensed 
tracks. 

What  pilots  can  do  as  a  result  of  their  air  track  knowledge  is 
assessed  by  arrays  tracking  whether  pilots  are  able  to  make  and  exe- 


Figure  2.4 

An  Array  Tracking  Pilots'  Knowledge  of  Fixed  Air  Tracks 
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cute  certain  types  of  decisions  referred  to  as  “advanced  tactics.”  We 
will  discuss  the  advanced  tactics  in  detail  later  in  the  briefing.  They 
are  based  on  interviews  with  pilots  who  participated  in  the  JTIDS 
experiments  (Hq.  USAF,  1997).  Importantly,  the  ability  to  perform 
these  advanced  tactics  is  associated  with  significant  improvements  in 
combat  effectiveness.  In  this  case  study,  we  model  quality  of  indi¬ 
vidual  decisionmaking  strictly  on  whether  pilots  have  sufficient 
awareness  to  run  the  advanced  tactics.  Thus,  this  case  study  effectively 
considers  only  the  roles  of  information  and  situational  awareness  and 
does  not  explicitly  consider  other  factors  that  can  have  major  impacts 
on  decisionmaking  (and  hence,  effectiveness) — combat  training,  C2 
structures,  team  interaction  factors,  and  so  on.  However,  we  believe 
these  additional  factors  are  similar  for  the  two  alternative  MCPs  con¬ 
sidered  in  this  case  study. 


CHAPTER  THREE 

Force  Characteristics 


The  NCO  CF  force  metrics  describe  information-  and  collaboration- 
related  components  of  the  joint  force  under  consideration.  In  this 
case  study,  the  force  metrics  comprise  descriptions  of  the  modeled  air 
superiority  force  packages  or  MCPs  and  their  information  sensors, 
voice  communications  systems,  and  data  communications  systems. 

The  Blue  force  packages  are  identical  except  for  the  presence  or 
absence  of  the  Link  16  data  communications  network.  Each  MCP 
includes  one  AWACS  aircraft  and  four  Blue  F-15s,  and  each  F- 1 5  has 
similar  Airborne  Moving-Target  Indicator  (AMTI)  radar,  noncoop¬ 
erative  target  recognition  (NCTR)  sensors,  an  identification,  friend- 
or-foe  (IFF),  transponder,  and  similar  weapon  systems.  The  four  are 
divided  into  two-ship  flights,  each  with  a  flight  lead  and  a  wingman. 


MCP  1:  Voice  Communications  Only 

The  first  MCP  (MCP  1)  is  equipped  with  only  voice  communica¬ 
tions  between  aircraft.  Figure  3.1  describes  a  typical  channel  structure 
for  voice  communications  used  by  Air  Force  fighters,  in  this  case  for 
an  MCP  with  four  fighters.  Each  pilot  monitors  two  separate  voice 
channels.  AWACS  broadcasts  aircraft  track  information  on  Channel 
1  to  the  Blue  aircraft.  The  four  fighter  aircraft  communicate  among 
themselves  on  Channel  2.  Each  fighter  can  listen  to  two  channels  at  a 
time,  but  only  one  aircraft  can  speak  on  a  channel  at  a  time. 
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The  Air  Force  has  developed  a  voice-coding  scheme  that  allows  pilots 
and  AWACS  flight  controllers  to  transmit  approximately  one  aircraft 
track  about  three  times  every  ten  seconds.  We  use  ten  seconds  as  the 
air  picture  track  update  cycle  time  because  this  is  the  rate  at  which  the 
AWACS  radar  antenna  rotates  or  performs  one  complete  surveillance 
cycle  of  the  battlespace.  So,  in  principle,  F-15  fighter  pilots  can 
receive  updated  air  track  information  from  AWACS  every  ten 
seconds,  if  AWACS  flight  controllers  have  the  time  to  verbally 
transmit  this  information  over  the  voice  network  every  radar  sweep 
and  if  there  is  "time  available”  to  transmit  the  information  over  the 
voice  channel  (i.e.,  if  there  is  no  contention  for  the  voice  channel).  By 
convention,  tracks  transmitted  later  than  ten  seconds  ago  are  consid¬ 
ered  to  be  invalid  or  “expired,”  on  the  grounds  that  the  tracked  plane 
may  have  moved  far  enough  in  ten  seconds  to  make  the  transmitted 
information  inaccurate. 

The  pilots  must  interpret  the  spoken  information  they  receive 
on  their  voice  radios  and  build  a  mental  “picture”  of  the  positions 

Figure  3.1 

Voice  Channel  Networks  for  MCP  1 

Channel  2:  intraflight  communications 
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and  velocities  of  reported  aircraft.  While  fighter  pilots  generally  have 
the  mental  ability  to  keep  air  track  information  in  their  mind  for  long 
periods,  the  utility  of  this  information  decreases  as  an  air  track  “age” 
grows.  An  air  track  with  an  age  of  ten  seconds  or  more  has  little 
utility  because  the  pilot  will  have  only  a  vague  idea  where  the  fast- 
moving  jet  fighter  may  be  (the  object  that  corresponds  to  the  air 
track),  especially  at  close  ranges.  We  approximate  the  process  of 
removing  old  information  from  a  fighter  pilot’s  mental  map  or 
“common  operational  picture”  of  the  battlespace  in  the  following 
way:  air  tracks  older  than  the  AWACS  update  rate  are  “dropped”  on 
the  grounds  that  the  tracked  plane  will  have  moved  far  enough  in  ten 
seconds  to  make  the  pilot’s  mental  air  track  position  and  velocity 
estimate  too  inaccurate  to  be  useful  in  a  high-speed  tactical  air  com¬ 
bat  engagement. 

The  interpretation  (and  mental  updating)  of  voice-delivered  air 
track  information  places  a  significant  cognitive  load  on  the  pilot.  This 
cognitive  load  must  be  handled  effectively  by  the  pilot  while  he  or  she 
simultaneously  performs  other  high-priority  mission-related  tasks. 
For  example,  in  addition  to  listening  to  and  interpreting  voice  chan¬ 
nel  traffic,  the  wingman  must  also  maintain  visual  contact  with  the 
flight  leader’s  aircraft,  fly  in  formation,  conduct  visual  surveillance  of 
the  battlespace — i.e.,  “check  six” — and  monitor  his  or  her  own  radar 
instruments.  Needless  to  say,  fighter  pilots  must  be  highly  trained  to 
effectively  conduct  all  these  tasks  simultaneously  or  in  rapid  succes¬ 
sion. 

For  the  sake  of  simplicity,  we  approximate  this  two-channel 
scheme  with  a  single  channel  in  which  one  pilot  is  allowed  to  speak  at 
a  time  and  all  other  aircraft  listen.  We  assume  that  AWACS,  on  aver¬ 
age,  uses  half  the  airtime,  and  that  the  four  fighter  aircraft  use  one- 
eighth  of  the  airtime  each.  While  not  optimal  (the  AWACS  broad¬ 
casting  all  the  time  is  near-optimal  because  it  has  all  tracks  and  the 
time  delay  in  transmission  negates  any  benefit  that  might  be  obtained 
by  transmitting  more  timely  tracks),  the  results  of  the  calculations 
vary  by  less  than  0.02  for  all  summary  metrics.  (Further  optimization 
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to  increase  the  metrics  is  possible,  such  as  giving  priority  to  transmit¬ 
ting  the  locations  of  the  Red  aircraft  that  are  only  held  by  the 
AWACS,  but  again  the  benefits  are  limited).  This  approach  has  limi¬ 
tations  near  the  performance  boundary  for  the  system,  particularly 
when  the  system  is  agile  (for  example,  if  operators  could  chose  to  give 
up  their  time  slot).  However,  it  was  used  in  the  case  study  because  it 
is  relatively  independent  of  the  MCP,  thus  enabling  later  compari¬ 
sons  with  other  MCPs.  Furthermore,  this  approximation  still  overes¬ 
timates  the  performance  of  the  voice  network  because  it  ignores  the 
impact  of  the  heavy  cognitive  loading  of  the  pilot  arising  from  the 
heavy  voice  channel  loading.  For  example,  it  ignores  the  impact  of 
pilots  frequently  having  to  listen  to  SAM  warning  information. 


MCP  2:  Link  16  Networking 

MCP  2  is  identical  to  MCP  1  except  for  the  addition  of  the  Link  16 
network.  Link  16  is  a  wireless  data  communications  system  that 
provides  air  track  and  other  information  to  fighter  aircraft,  other 
weapons  platforms,  and  C2  nodes  equipped  with  JTIDS  and  Multi¬ 
functional  Information  Distribution  System  (MIDS)  communica¬ 
tions  terminals.  Link  16  employs  a  time-division  multiple-access 
(TDMA)  wireless  network  structure  and  a  jam-resistant  frequency- 
hopping  waveform.  The  Link  16  networking  structure  is  illustrated 
Figure  3.2.  As  shown,  the  topology  of  a  Link  16  network  is  entirely 
different  from  that  of  voice  networks.  In  this  type  of  network,  each 
participant  (or  network  node)  can  receive  all  transmissions  made  by 
other  network  participants  in  the  same  broadcast  region. 

Fighters  equipped  with  Link  16  can  receive  air  track  information 
from  neighboring  fighter  aircraft  and  from  AWACS  (if  AWACS  is 
within  a  line  of  sight  of  the  fighter).  A  Link  16  network  is  composed 
of  128  time  slots  per  second,  with  each  slot  capable  of  describing  a 
single  airplane  track  to  a  high  degree  of  accuracy.  Each  Blue  plane 
(and  other  information  sources,  such  as  those  broadcasting  SAM 
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threats)  is  allocated  one  or  more  time  slots  during  which  they  can 
transmit  information.  Link  16  air  tracks  received  by  a  particular 
fighter  from  other  aircraft  are  shown  on  a  display  screen  in  the  cock¬ 
pit  along  with  air  tracks  detected  by  the  aircraft’s  organic  sensors. 
Therefore  each  fighter  pilot  in  a  Link  16  equipped  MCP  can  display 
nearly  the  same  air  track  information  or  the  same  picture  of  the  bat- 
tlespace. 

Figure  3.2 

The  Link  16  Network  for  MCP  2 


Link  16  network 
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CHAPTER  FOUR 


Quality  of  Organic  Information 


Input  Factors  and  Specific  Metrics 

In  general,  the  computation  of  quality  of  organic  information  (here, 
the  aircraft  track  information  detected  by  each  plane’s  sensors)  is 
complicated.  Each  Blue  aircraft  is  equipped  with  multiple  sensors, 
each  with  distinct  capabilities,  such  AMTI  radar,  Identification, 
Friend,  Foe,  or  Neutral  (IFFN)  transponders,  NCTR  systems,  and  so 
on.  The  computation  of  information  quality  could  take  into  account 
a  wide  range  of  factors  for  each  sensor,  such  as  sensor  coverage  statis¬ 
tics  (field  of  regard  and  revisit  rate),  probability  of  track  detection 
within  sensor  field,  probability  of  false  alarms,  probability  of  classifi¬ 
cation  error,  sighting  location  and  velocity  errors,  and  so  on.  Figure 
4.1  provides  a  more  complete  list  of  sensor  attributes  that  might  be 
important  to  consider  when  calculating  or  estimating  the  quality  of 
organic  information  for  individual  platforms. 

However,  because  the  purpose  of  this  report  is  to  compare  voice- 
only  and  Fink  16  air  superiority  MCPs,  it  is  possible  to  simplify  the 
analysis  greatly.  The  sensors  in  each  MCP  are  identical,  and  the  sce¬ 
narios  for  which  the  metrics  are  calculated  are  also  the  same.  We  sim¬ 
ply  declare  an  organic  information  position  common  to  both  MCPs 
(in  terms  of  what  each  sensor  reports)  and  conduct  measurements 
from  that  position.  We  use  the  “threat  information  position”  to  com¬ 
pute  partial  scores  for  the  four  air-track  quality  metrics  discussed  in 
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Figure  4.1 

Factors  Potentially  Involved  in  Computing  Quality  of  Organic  Information 
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Chapter  Two:  Completeness:  Detection  (of  air  tracks),  Correctness: 
ID,  Accuracy:  Location,  and  Accuracy:  Velocity. 


Inputs,  Calculations,  and  Individual  Platform  Results 

Figure  4.2  shows  the  organic  information  position  of  each  Blue  air¬ 
craft  with  respect  to  what  they  know  about  the  positions  of  the  Red 
aircraft  (the  “threat  tracks”).  From  the  figure,  AWACS  detects  the 
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Figure  4.2 

Quality  of  Organic  Information  for  Threat  (Red)  Tracks 


Red  1,  2 


Red  3,  4 


Q]  Completeness:  detection 

Qt3  Accuracy:  location 

Qt2  Correctness:  ID 

Qt4  Accuracy:  velocity 

position  of  all  four  Red  aircraft  within  the  battlespace.  Two  of  the 
Blue  fighters  (Blue  1 1  and  Blue  1 2)  have  detected  two  of  the  Red  air¬ 
craft  with  their  radar;  the  other  two  Blue  fighters  do  not  have  any 
organic  Red  aircraft  track  information. 

As  noted  in  Chapter  Two,  the  currency  of  track  updates  directly 
impacts  the  metric  scores.  Table  4.1  scores  the  organic  track  reports 
by  the  age  of  the  report.  Tracks  with  a  latency  of  less  than  a  second 
from  sensor  detection  to  pilot  report  are  considered  near  real  time. 
Because  these  tracks  are  sufficient  for  making  precision  maneuver  and 
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Table  4.1 

Table  of  Metric  Scores  by  Track  Latency 


Metric  Score  for  a  Track  Updated  Within  a  Given  Time  Range 

Less  Than  One 

One  to  Ten 

More  Than  Ten 

Metric 

Second 

Seconds 

Seconds 

Detection  or  ID 

1.00 

0 

Location  or  velocity 

1.00 

0 

targeting  decisions,  they  score  the  maximum  value  of  1.0  for  all  four 
metrics;  we  assume  that  the  Blue  force’s  radars  and  detection  sensors 
are  highly  likely  to  be  correct.  Tracks  with  a  latency  of  one  to  ten  sec¬ 
onds  are  considered  “cue  only,”  suitable  for  general  awareness  and 
cuing,  but  not  for  precision  maneuver  or  targeting.  They  score  1 .0  for 
detection  and  ID  correctness  but  only  score  0.25  for  location  and 
velocity  correctness.  That  is,  because  the  measures  of  accuracy  are 
assessed  over  time,  a  separate  assessment  of  timeliness  is  not  required. 
Tracks  with  a  time-to-use  of  more  than  ten  seconds  score  zero  for  all 
four  metrics,  because  air  tracks  with  this  time  latency  are  ascribed  to 
have  little  utility  for  close  in  air-to-air  combat.1 

The  small  graphs  on  Figure  4.2  show  the  resulting  quality  of 
organic  information  for  threat  air  tracks.  Each  graph  shows  the  aver¬ 
age  metric  score  for  all  four  Red  aircraft  as  detected  by  each  Blue 
plane.  AWACS  detects  all  four  Red  aircraft  in  the  battlespace,  but  has 
a  ten-second  radar  sweep  delay.  An  aircraft  will  be  detected  once  each 
sweep  and  then  not  detected  again  for  another  ten  seconds  (on  aver¬ 
age).  That  is,  most  of  the  time  the  AWACS  track  is  more  than  one 
second  old,  meaning  it  scores  1.0  for  detection  completeness  and  ID 
correctness  but  only  0.25  for  location  and  velocity  correctness.  Blue 
11  and  Blue  12  each  detect  two  of  the  four  Red  aircraft  with  near- 
real  time  precision  (provided  by  the  F-15  sensors),  so  they  each  score 
0.5  for  all  four  metrics.  Finally,  Blue  13  and  Blue  14  each  detect  none 
of  the  four  Red  aircraft,  so  they  each  score  zero  for  all  four  metrics. 


1  Such  high-latency  air  tracks  may  have  some  utility  for  general  threat  warning  information, 
but  more  precise  threat  location  and  heading  information  is  needed  in  air-to-air  combat. 
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Figure  4.3  describes  the  quality  of  organic  information  with 
respect  to  each  Blue  aircraft’s  knowledge  of  the  other  Blue  aircraft  in 
the  battlespace.  Each  graph  now  shows  the  average  metric  score  for  all 
five  Blue  aircraft  as  detected  by  each  Blue  plane.  Thus,  AWACS 
knows  its  own  track  exactly  (via  GPS)  and  knows  the  track  of  each 
Blue  fighter  within  a  ten-second  sweep  windows.  It  thus  scores  1.0 
for  detection  completeness  and  ID  correctness,  and  0.4  for  location 
and  velocity  correctness.  Each  Blue  fighter  knows  its  own  track 
exactly  but  does  not  know  the  track  of  any  other  Blue  aircraft  organi¬ 
cally;  each  thus  scores  0.2  for  all  four  metrics. 

Figure  4.3 

Quality  of  Organic  Information  for  Blue  Tracks 
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Overall  Results 

We  average  across  all  scores  for  all  pilots’  organic  awareness  of  both 
Red  and  Blue  tracks,  to  get  an  overall  score  of  0.28  for  quality  of 
organic  information  (also  shown  in  Figure  4.4).  This  value  applies  to 
both  the  voice-only  and  Link  1 6  networks  because  both  start  with  the 
same  organic  information  position. 

Figure  4.4 

Overall  MCP  Scores  for  Quality  of  Individual  Information 
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CHAPTER  FIVE 

Degree  of  Networking 


Specific  Metrics 

The  NCO  CF  identifies  four  attributes  associated  with  the  top-level 
degree  of  networking  concept:  reach,  quality  of  service,  network 
assurance,  and  agility.  The  latter  two,  in  turn,  have  subsidiary  attri¬ 
butes;  of  these  we  selected  availability,  integrity/privacy,  and  network 
adaptation  for  this  analysis.  For  each,  we  directly  translate  the  func¬ 
tionalities  of  the  voice-only  and  Link  16  networks,  as  given  by  their 
technical  specifications,  into  metric  scores. 


Inputs,  Calculations,  and  Individual  Platform  Results 

Table  5.1  presents  the  resulting  five  degree  of  networking  metrics  for 
each  of  five  communications  modes  for  MCP  1  (voice  only). 

For  the  first  MCP,  voice  only,  voice  is  the  only  mode  supported. 
The  NCO  CF  defines  “reach”  as  the  degree  to  which  nodes  can  con¬ 
nect  and  communicate,  and  the  metric  is  the  percentage  of  nodes  that 
can  communicate  in  the  desired  format.  Because  (in  our  single  voice- 
channel  scenario)  each  plane  can  transmit  on,  and  each  plane  listens 
to,  the  channel,  we  define  the  value  of  reach  for  this  mode  to  be  1.0. 

The  NCO  CF  defines  “quality  of  service”  as  the  ability  to  pro¬ 
vide  communications  and  storage  services  and  a  vector  of  metrics 
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Table  5.1 

Degree  of  Networking  Metrics  for  MCP  1  (Voice  Only) 


Attribute 

Voice 

Messag¬ 

ing 

Data 

Image 

Video 

Reach 

1 

0 

0 

0 

0 

Each  plane  can  only 

(none) 

(none) 

(none) 

(none) 

Quality  of  service 

listen  to  one  mes¬ 
sage  at  a  time 

0.012 

0 

0 

0 

0 

(2  MIL  vocoder 

(none) 

(none) 

(none) 

(none) 

Network  assurance 

channels) 

(Rate:  0.3  tracks/ 
sec.;  1 7  tracks/sec. 
max) 

(70%  of  tracks 
heard  correctly) 

~1 

0 

0 

0 

0 

(availability) 

(none) 

(none) 

(none) 

(none) 

Network  assurance 

~1  (jam-resistant, 

0 

0 

0 

0 

(integrity/ 

frequency-hopping 

(none) 

(none) 

(none) 

(none) 

privacy) 

Network  adapta¬ 

military  encryption) 

~1 

0 

0 

0 

0 

tion 

(none) 

(none) 

(none) 

(none) 

NOTE:  Average  score  (across  five  mission  modes,  five  attributes)  =  0.16.  Metric  scores 
of  zero  are  assigned  for  some  metrics  in  rows  two  through  five  because  the  voice 
network  lacks  messaging,  data,  image,  or  video  capabilities. 


including  bandwidth,  packet  delay,  delay  jitter,  and  data  loss.  In  this 
case  study,  we  are  only  interested  in  communications  and  focus  on 
bandwidth  and  data  loss.  Also,  since  we  want  metrics  normalized 
between  zero  and  one  for  this  case  study,  quality  of  service  equals  the 
effective  transmission  rate  of  the  network  divided  by  the  ideal  trans¬ 
mission  rate.  The  effective  transmission  rate  is  equal  to  the  physical 
transmission  rate  multiplied  by  the  probability  that  transmitted  tracks 
are  received  correctly.  We  track  both  of  these  components  in  order  to 
determine  the  metrics  for  information  share-ability. 

The  Air  Force  has  developed  a  scheme  allowing  speakers  to 
transmit  three  tracks  every  ten  seconds,  which  becomes  0.3  tracks  per 
second.  According  to  the  pilots  we  interviewed,  the  standard  military 
vocoder  equipment  can  frequently  be  hard  to  hear,  such  that  on  aver¬ 
age  pilots  hear  only  about  70  percent  of  the  transmitted  information 
(the  rest  is  garbled).  There  are  17  sensed  tracks  in  the  scenario,  and 
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ideally  each  should  be  updated  at  least  once  a  second,  making  the 
ideal  transmission  rate  17  tracks  per  second.  Performing  the  calcula¬ 
tion  yields  a  measurement  of  just  0.012  for  quality  of  service. 

For  network  assurance,  we  consider  the  availability,  integrity, 
and  privacy  subattributes.  Availability  reflects  the  reliability  of  the 
network,  even  in  the  face  of  Red  force  jamming  and  environmental 
obstacles.  Integrity  and  privacy  reflect  the  protection  of  the  network 
from  Red  force  surveillance,  so  we  combine  these  into  a  single  metric. 
(The  other  assurance  subattributes — authenticity  and  nonrepudia¬ 
tion — were  not  considered  because  in  a  broadcast  network  without 
data  storage  they  are  based  on  privacy  and  integrity.)  Both  assurance 
metrics  are  constants  in  this  model,  set  at  1.0  because  the  AN/ARC- 
164  Have  Quick  radio  and  vocoder  system  are  highly  reliable  in 
flight,  and  frequency-hopping  and  encryption  are  provided.  We  also, 
for  simplicity’s  sake,  assume  a  permissive  environment  without  Red 
force  jamming  and  surveillance  and  without  environmental  obstacles. 

Network  adaptability  refers  to  the  ability  of  the  network  (and 
associated  concepts  of  operation)  to  support  changes  to  the  network, 
participants  using  the  network,  and  processes  during  operation.  The 
voice-only  network  is  relatively  adaptable  during  the  mission  (channel 
frequencies  are  preset  in  advance  of  the  mission  and  can  be  changed 
during  the  mission  to  a  preassigned  backup  frequency),  so  the  score 
for  adaptation  is  approximately  one. 

The  average  score  for  the  voice-only  MCP  across  all  five  attri¬ 
butes  and  five  modes  is  0.16. 

Table  5.2  presents  the  five  networking  metrics  scores  for  each  of 
five  communications  modes  for  MCP  2  (voice  plus  Link  16). 

In  contrast  to  voice  only,  the  Link  16  and  voice  MCP  supports 
multiple  communications  formats:  voice,  messaging,  data,  and  image 
transmission  (only  video  is  not  currently  supported).  Thus,  the  Link 
16  MCP  scores  a  value  of  1.0  for  reach  for  all  modes  except  video. 

The  Link  16  MCP  scores  a  value  of  1.0  for  quality  of  service  for 
messaging,  data,  and  images.  The  Link  16  network  has  a  data  trans- 


Table  5.2 

Degree  of  Networking  Metrics  for  MCP  2  (Link  16) 


Attribute 

Voice 

Messaging 

Data 

Image 

Video 

Reach 

1 

(2  transmit,  rest 
receive  only) 

1 

(up  to  30  nets,  128 
participants  per  net) 

1 

(up  to  30  nets,  128 
participants  per  net 

1  (most  receive  only) 

0  (none) 

Quality  of  service 

0.012 

2  MIL  vocoder 

channels 

(3  tracks/10  sec.) 

1 

28.8  to  238  Kbps 
(128  tracks/sec.) 

JTIDS  Waveform 

Tadil  Ja 

Depends  on  IA  level, 
number  of  nodes 

1 

28.8  to  238  Kbps 
(128  tracks/sec.) 

JTIDS  Waveform 
Limited  Free  text 
Depends  on  IA  level, 
number  of  nodes 

1 

28.8  to  238  Kbps 
(128  tracks/sec.) 

JTIDS  Waveform 
Depends  on  IA  level, 
number  of  nodes 

0  (none) 

Network  assurance 
(availability) 

~1 

1 

1 

1 

NA 

Network  assurance 
(integrity/privacy) 

~1  (basic  military 
encryption) 

1  (basic  military 
encryption) 

AJ  waveform 

1  (basic  military 
encryption) 

AJ  waveform 

1  (basic  military 
encryption) 

AJ  waveform 

NA 

Network  adaptation 

1 

0.5 

(depends  on  service) 

0.5 

(depends  on  service) 

0.5 

(depends  on  service 
and  terminal) 

NA 

aTactical  Digital  Information  Link  J  (TADIL  J)  message  catalog  contains  256  message  types.  TADIL  J  messages  are  composed  of  one  or 
more  code  words,  each  7t  bits  (7-  data,  5  parity). 

NOTE:  Average  score  (across  five  transmission  modes,  five  attributes)  =  0.74. 
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mission  rate  of  128  tracks  per  second,  which  exceeds  the  ideal  trans¬ 
mission  rate  of  17  tracks  per  second.  (More  strictly,  the  bandwidth 
devoted  to  each  plane — 64  tracks  per  second  for  AWACS,  16  tracks 
per  second  for  each  Blue  fighter — greatly  exceeds  the  number  of 
tracks  each  blue  aircraft  must  transmit.)  The  Link  16  network  also 
transmits  these  tracks  with  a  high  degree  of  accuracy,  that  is,  with  no 
data  loss.  Thus  the  quality  of  service  for  data,  messaging,  and  image 
transmission  is  set  to  1.0.  The  MCP  scores  a  value  of  0.012,  though, 
for  voice,  since  the  Link  16  system  retains  the  same  type  of  vocoders 
and  constraints  as  in  the  voice-only  MCP  if  air  track  data  are  trans¬ 
mitted  only  by  voice  communications.  The  Link  16  MCP  scores  1.0 
for  availability  (network  is  highly  reliable  and  also  provides  antijam 
capabilities  because  of  its  frequency-hopping  wave  form)  and  1.0  for 
integrity  and  privacy  (network  is  encrypted).  (We  note  that  the  quan¬ 
titative  benefits  and  trade-offs  that  accrue  from  Link  16’s  use  of  fre¬ 
quency-hopping  to  prevent  jamming  are  not  assessed  in  this  analysis.) 
Finally,  there  is  some  ability  to  customize  the  Link  16  network 
dynamically  for  the  messaging,  data  and  image  modes,  so  we  set  the 
Network  Adaptability  scores  for  these  modes  at  0.5. 


Overall  Results 

•  The  average  score  for  degree  of  networking,  across  all  metrics 
and  modes  for  the  voice-only  MCP  was  0.16. 

•  In  comparison,  the  average  score  across  all  metrics  and  modes 
for  the  Link  16  MCP  was  0.74.  Figure  5.1  (next  page)  compares 
the  two. 
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Figure  5.1 

Overall  MCP  Scores  for  Degree  of  Networking 
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CHAPTER  SIX 


Degree  of  Information  "Share-Ability" 


The  computation  of  metrics  that  depend  on  other  metrics  is  a  com¬ 
plicated,  multistage  process.  We  describe  this  process  for  degree  of 
information  “share-ability”  below,  which  depends  on  the  subsidiary 
measurements  for  quality  of  organic  information  and  degree  of  net¬ 
working. 


Specific  Metrics 

The  NCO  CF  specific,  subsidiary  metric  calculated  for  degree  of 
information  “share-ability”  is  quantity  of  retrievable  information, 
which  assesses  the  ability  of  aircraft  to  retrieve  air  track  information 
from  the  MCPs’  networks.  For  this  case  study,  this  metric  is  the  array 
of  probabilities  providing,  for  each  of  the  17  sensed  air  tracks  in  the 
battlespace,  the  probability  that  a  Blue  aircraft  has  “retrieved” 
(received)  an  update  on  that  track  within  a  given  time  period  (either 
less  than  one  second  or  one  to  ten  seconds;  recall  that  these  two 
latency  bands  directly  impact  the  calculation  of  the  quality  of  infor¬ 
mation  measurements). 

Ordinarily,  to  calculate  the  quantity  of  retrievable  information, 
one  would  need  to  calculate  another  metric,  the  quantity  of  posted 
information  (which  assesses  the  ability  of  aircraft  to  post  air  track 
information  to  the  MCPs’  networks),  first.  Fiowever,  in  this  report, 
we  can  simplify  the  analysis,  as  all  air  tracks  “posted”  (broadcasted)  is 
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immediately  “retrieved”  (received)  by  the  other  aircraft  in  rhe  MCPs’ 
wireless  networks.  The  only  distinction  is  whether  some  of  the  broad¬ 
cast  data  is  lost  during  transmission.  Thus,  we  need  only  calculate  a 
single  metric. 


Inputs 

Figure  6.1  summarizes  the  inputs  contributing  to  the  computation  of 
the  degree  of  information  “share-ability”  measurements.  As  shown, 
some  of  the  inputs  are  derived  from  the  earlier  degree  of  networking 
measurements,  while  others  are  exogenous. 

Figure  6.1 

Factors  Impacting  Quality  of  Retrievable  Information 


Degree  of  networking  inputs  Exogenous  inputs 
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Table  6.1  provides  a  fuller  description  of  the  inputs  used  to 
compute  quantity  of  retrievable  information  as  well  as  what  those 
inputs  are  for  both  the  voice-only  and  Link  16  MCPs. 

Of  interest  in  Table  6.1,  for  the  voice-only  MCP,  is  that  the  Air 
Force  has  standards  through  which  three  tracks  can  be  transmitted 
over  voice  every  ten  seconds.  This  implies  that  all  voice-transmitted 
posts  will  be  at  best  in  the  one-  to  ten-second  latency  band.  Second, 
note  that  pilots  only  “hear”  about  70  percent  of  the  transmitted 
tracks.  For  Link  16,  the  data  transmission  rate  is  up  to  128  track 
updates  per  second,  loss  free  (assuming,  as  the  case  study  does,  a  per¬ 
missive  environment). 


Calculations 

Below,  we  describe  the  steps  involved  in  calculating  quantity  of 
retrievable  information  and  how  these  steps  apply  to  the  voice-only 
and  Link  16  MCPs.  We  first  compute  maximum  potential  transmis¬ 
sion  rates  for  air  tracks,  then  use  these  rates  to  compute  the  probabili¬ 
ties  that  particular  sensed  air  tracks  will  be  “retrieved.”  As  might  be 
expected,  the  resulting  steps  involve  a  series  of  product  calculations. 
The  steps  occur  in  three  phases. 

Phase  1 :  calculate  the  maximum  rate  at  which  the  Blue  aircraft 
can  share  air  tracks  across  the  MCPs’  wireless  networks. 

1.  If  nodes  can  transmit  to  the  network,  go  to  step  2.  If  not,  the 
maximum  rate  is  0. 

a.  All  aircraft  in  both  MCPs  can  transmit. 

2.  Multiply  data  transmission  rate  by  accuracy  to  get  maximum 
rate. 

a.  For  voice-only:  have  3  tracks  per  10  seconds  times  70  percent 
accuracy,  which  equals  2.1  tracks  per  second. 

b.  For  Link  16:  have  128  tracks  per  second,  at  100  percent  accu¬ 
racy. 
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Table  6.1 

Inputs  to  Quantity  of  Retrievable  Information 


Input  Input  for  Voice  Only  Input  for  Link  16 

From  Quality  of  Organic  Information 

List  of  information  ele-  17  sensed  air  tracks  Same  as  voice  only 

ments  to  be  retrieved 
from  the  network 


Reach:  whether  and  how 
nodes  can  transmit  from 
and  to  the  network 
Quality  of  service:  data 
transmission  rates  and 
accuracy 


Network  assurance:  prob¬ 
ability  of  service  provi¬ 
sion,  lack  of  data 
corruption 

Network  agility:  prob¬ 
ability  that  quality  of 
service  will  be  delivered 
in  variety  of  mission  envi¬ 
ronments 


From  Degree  of  Networking 

AWACS,  fighters  can  trans¬ 
mit  track  info  over  voice 

USAF  standards  for  voice 
channels:  maximum  of 
three  tracks  every  ten  sec¬ 
onds. 

From  pilot  interviews, 
about  30%  of  track  info  is 
lost  (not  heard  correctly) 
using  voice-only  channels. 

Assume  permissive  environ¬ 
ment,  so  service  provision 
is  100%  and  data  corrup¬ 
tion  is  0% 

Assume  constant  environ¬ 
ment  for  case  study  (no 
change) 


AWACS,  fighters,  can 
transmit  track  info 
digitally  over  Link  16 
Link  16  has  128  time 
slots  per  second;  each 
can  provide  a  track 
update. 

In  assumed 
permissive 
environment,  no 
track  info  is  lost. 

Same  as  voice  only 


Same  as  voice  only 


Number  of  network  nodes 
attempting  to  retrieve 
Number  and  size  of  data 
elements  to  be  retrieved 
Policies  determining  prior¬ 
ity  for  retrieval 


Lifespan:  Length  of  time 
for  which  received  tracks 
are  considered 
"retrieved" 


Exogenous  Variables 

5  (AWACS,  4  Blue  fighters) 

17  sensed  "air  tracks" 

AWACS  has  50%  of  airtime; 
each  Blue  fighter  has  a 
12.5%  share  of  air  time. 
Red  tracks  have  two-thirds 
of  air  time;  Blue  tracks 
have  one-third  of  air  time. 

Top  band:  latency  under  1 
second 

Second  band:  latency  be¬ 
tween  1  andIO seconds 


Same  as  voice  only 

Same  as  voice  only 

AWACS  has  50%  of 
time  slots;  each  Blue 
fighter  has  a  12.5% 
share  of  time  slots. 
Red  tracks  have  2/3  of 
time  slots;  Blue  tracks 
have  1/3  of  time  slots. 

Same  as  voice  only 
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Phase  2:  calculate  the  maximum  number  of  tracks  of  each  type 
(Red  or  Blue,  particular  aircraft)  that  the  Blue  aircraft  may  have 
“retrieved”  from  the  network  at  any  time. 

3.  Multiply  maximum  rate  by  lifespan  to  get  maximum  number  of 
“retrieved”  tracks. 

a.  For  voice  only:  0  tracks  “retrieved”  within  one-second  band; 
2.1  tracks  within  one-  to  ten-second  band. 

b.  For  Link  16:  128  tracks  retrieved  within  one-second  band; 
1,280  tracks  within  one-  to  ten-second  band. 

4.  Multiply  maximum  number  of  retrieved  tracks  by  the  percent¬ 
age  of  airtime  dedicated  to  each  type  of  track  and  by  the  per¬ 
centage  of  airtime  devoted  to  each  aircraft.  This  yields  the 
expected  number  of  tracks  of  each  type  that  can  be  “retrieved”  at 
any  time. 

a.  For  voice-only:  We  look  only  at  the  1-10  second  band.  (1) 
AWACS/Red  tracks:  (2.1  maximum  tracks  retrieved)  (50  per¬ 
cent  of  airtime  for  AWACS)  (two-thirds  of  airtime  for  Red 
tracks)  =  average  of  0.7  tracks  in  one-  to  ten-second  band.  (2) 
Fighters/Red  tracks:  (2.1  tracks  retrieved) (12. 5  percent  of  air¬ 
time)  (two-thirds  of  airtime  for  Red)  =  0.175  tracks  in  one-  to 
ten-second  band.  (3)  AWACS/Blue  tracks:  using  one-third  of 
airtime  for  Blue  tracks,  get  0.35  tracks  in  one-  to  ten-second 
band.  (4)  Fighters/Blue  tracks:  0.0875  tracks  in  one-  to  ten- 
second  band. 

b.  For  Link  16:  We  look  only  at  the  less-than-one-second  band 
because  of  the  high  transmission  speed.  (1)  AWACS/Red 
tracks:  (128  maximum  tracks  retrieved)  (50  percent  of  airtime 
for  AWACS)  (two-thirds  of  airtime  for  Red  tracks)  =  average 
of  42.67  tracks  in  less-than-one-second  band.  (2)  Fighters/Red 
tracks:  (2.1  tracks  retrieved) (12. 5  percent  of  airtime)(two- 
thirds  of  airtime  for  Red)  =  10.67  tracks  in  less-than-one- 
second  band.  (3)  AWACS/Blue  tracks:  using  one-third  of 
airtime  for  Blue  tracks,  get  21.33  tracks  in  one-  to  ten-second 
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band.  (4)  Fighters/Blue  tracks:  5.33  tracks  in  less-than-one- 
second  band. 

Phase  3 :  calculate  the  probability  that  a  given  air  track  will  be 
“retrieved”  at  any  time. 

5.  Divide  the  expected  “retrieved”  tracks  of  each  type  by  the  num¬ 
ber  of  air  tracks  of  each  type  to  retrieve.  If  this  number  is  more 
than  one,  probability  track  has  been  retrieved  is  100  percent. 
Otherwise,  probability  is  this  number.  (For  results  for  each  of 
the  17  sensed  air  tracks  by  MCP,  see  below.) 

For  the  sake  of  simplicity,  the  calculations  above  did  not  include 
the  input  factors  not  directly  included  in  the  case  study,  such  as  net¬ 
work  assurance  and  network  agility.  Fiad  these  factors  been  included, 
they  would  have  been  treated  in  a  similar  manner — for  example,  mul¬ 
tiplying  the  track  transmission  rates  by  an  additional  factor  repre¬ 
senting  the  probability  that  tracks  are  not  jammed  or  compromised. 


Individual  Platform  Results 

Figure  6.2  shows  the  total  probabilities  of  retrieving  air  tracks  for 
each  plane  in  the  Blue  force,  using  the  procedures  discussed  above. 
We  represent  the  probabilities  of  retrieving  tracks  in  the  both  less- 
than-one-second  and  one-  to  ten-second  timeliness  bands.  We  need 
this  level  of  differentiation  to  properly  compute  the  quality  of  infor¬ 
mation  metrics. 

For  the  voice-only  network,  a  fairly  small  percentage  of  the  total 
tracks  are  expected  to  be  retrieved  at  any  time  in  the  one-  to  ten- 
second  band  (about  12  percent).  No  tracks  are  posted  in  the  less- 
than-one-second  band.  Conversely,  the  Link  16  network  posts  100 
percent  of  the  available  track  information  within  the  less-than-one- 
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Figure  6.2 

Percentage  of  Information  Retrieved  by  Timeliness  Band 
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second  latency  band,  although  the  AWACS  tracks  were  only  obtained 
within  the  one-  to  ten-second  interval.  Averaging  the  quantity  of 
retrieved  information  across  all  Blue  aircraft,  the  voice  network  man¬ 
ages  only  an  overall  score  of  0.08  for  degree  of  information  “share- 
ability,”  whereas  the  Link  16  network  has  a  perfect  score  of  1.0. 


Overall  Results 

Averaging  the  quantity  of  retrieved  information  across  all  Blue  air¬ 
craft  yields: 
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•  The  voice-only  MCP  manages  only  an  overall  score  of  0.08, 
because  of  the  very  limited  number  of  track  reports  that  can 
actually  be  carried  over  the  voice  channel. 

•  The  Link  16  MCP  has  a  perfect  score  of  1.0  because  all  sensed 
air  tracks  are  immediately  reported  to  the  other  Blue  aircraft  via 
the  Link  1 6  network. 

Figure  6.3  compares  these  two  scores. 

Figure  6.3 

Comparing  MCP  Scores  for  Degree  of  Information  "Share-Ability" 
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CHAPTER  SEVEN 


Quality  of  Individual  Information 


Specific  Metrics 

In  this  case  study,  the  quality  of  individual  information  comprises 
four  subsidiary  metrics  about  the  air  tracks:  completeness  of  detection 
of  air  tracks,  correctness  of  ID,  accuracy  of  location,  and  accuracy  of 
velocity.  All  four  depend  on  the  same  inputs:  the  initial  quality  of  the 
air  tracks  (quality  of  organic  information),  quality  changes  incurred 
when  sharing  the  air  tracks  over  the  network  (degree  of  information 
“share-ability”),  and  several  other  exogenous  factors.  All  four  have  the 
same  form.  They  are  all  arrays  with  each  entry  presenting  the  quality 
measurement  on  a  Blue  aircraft’s  knowledge  of  an  air  track.  In  par¬ 
ticular,  each  measurement  is  the  probability  of  “knowing”  the  unique 
track  to  within  a  particular  latency  and,  weighted  by  the  value  of  that 
knowledge.  Recall  that,  for  location  and  velocity  metrics,  knowing  a 
track  to  within  less  than  one  second  has  value  1.0  and  that  knowing  a 
track  to  within  1-10  seconds  has  value  0.25. 

In  comparison  to  degree  of  information  “share-ability,”  how¬ 
ever,  of  interest  here  is  each  aircraft’s  knowledge  of  the  nine  unique  or 
fixed  tracks  in  the  battlespace  (AWACS,  four  Blue  fighters,  four  Red 
fighters),  not  the  17  sensed  tracks.  Thus,  our  calculations  will  account 
for  the  fusion  of  the  sensed  tracks  into  representations  of  the  unique 
tracks. 
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Inputs 

Figure  7.1  summarizes  the  inputs  to  quality  of  individual  informa¬ 
tion,  indicating  which  are  derived  from  the  quality  of  organic  infor¬ 
mation  measurements  or  the  degree  of  information  “share-ability” 
measurements  and  are  exogenous. 

Table  7.1  describes  the  inputs  used  to  compute  the  quantity  of 
individual  information  metrics,  as  well  as  what  those  inputs  are  for 
both  the  voice-only  and  Link  16  MCPs.  Of  interest  in  Table  7.1  are 
the  fusion  formulas.  In  this  report,  we  assume  that  fusion  is  equiva¬ 
lent  to  selecting  the  best-known  sensed  track  to  represent  the  corre¬ 
sponding  unique  track.  This  tends  to  overestimate  the  quality  of 
fusion  for  voice-only  networks  (as  all  fusion  must  be  done  mentally) 
and  for  Link  16  Red  tracks  (also  requires  mental  fusion,  although 


Figure  7.1 

Factors  Affecting  Quality  of  Organic  Information 


Quality  of  organic  information 


Exogenous 

inputs 


Degree  of  information 
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graphical  displays  of  tracks  help  greatly);  it  tends  to  underestimate  the 
quality  of  fusion  for  Link  16  Blue  tracks  because  it  ignores  the 
improvements  in  accuracy  resulting  from  the  ability  to  average  multi¬ 
ple  Blue  aircraft  tracks.  However,  we  believe  the  formulas  are  reason¬ 
able  approximations  based  on  our  pilot  interviews,  especially  for  an 
illustrative  case  study. 

Table  7.1 

Inputs  to  Quality  of  Individual  Information  Metrics 


Input 

Input  for  Voice  Only 

Input  for  Link  16 

From  Quality  of  Organic  Information 

Array  providing 
organic  quality 
measurements  for 
the  17  sensed  air 
tracks 

See  Chapter  Four 

See  Chapter  Four 

From  Degree  of  Networking 

Array  providing  quan¬ 
tity  of  information 
retrieved  measure¬ 
ments  for  the  17 
sense  air  tracks 

See  Chapter  Six 

See  Chapter  Six 

Exogenous  Variables 

Parameters  describing 
the  fusion  of  the 
sensed  air  tracks 

No  automated  fusion 
performed;  pilots  must 
perform  mental  fusion 
(are  trained  to  do  so). 
Pilots  trained  to  use 
organic  information 
(visually  displayed  on 
radar  screen)  in  prefer¬ 
ence  to  radio-reported 
information. 

Resulting  formula: 
knowledge  of  a  unique 
track  is  maximum  of 
what  is  known  about 
any  corresponding 
sensed  track  (whether 
organic  or  over  voice- 
only  network). 

Link  16  fuses  Blue  track  posi¬ 
tions  automatically;  does  not 
fuse  Red  tracks. 

However,  Red  tracks  of  the 
same  aircraft  sensed  by  fight¬ 
ers  (which  have  real-time 
tracking  capabilities)  will 
appear  joined  as  a  single  dot; 
Red  tracks  sensed  by  AWACS 
have  a  different  symbol, 
which  pilots  are  trained  to 
acknowledge. 

Resulting  formula:  knowledge 
of  a  unique  track  is  maximum 
of  what  is  known  about  any 
corresponding  sensed  track 
(whether  organic  or  over  Link 

16  network). 
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Calculations 

Calculating  the  quality  of  individual  information  measurements  has 
two  phases.  The  first  represents  fusion,  in  which  we  calculate  the 
probability  that  each  Blue  aircraft  has  knowledge  of  one  of  the  nine 
unique  tracks  in  a  particular  latency  band.  The  second  uses  the  prob¬ 
abilities  to  calculate  the  four  quality  measurements  for  each  track. 

Phase  1:  Calculate  the  probability  of  “knowing”  a  unique  air 
track  within  a  particular  latency  band  (less  than  one  second  or  one  to 
ten  seconds). 

1.  Create  a  new  array  whose  entries  reflect  the  best  of  source  of 
information  each  Blue  aircraft  has  on  a  particular  sensed  air 
track  (organic  or  retrieved  from  the  network). 

a.  For  voice  only,  best  source  of  information  each  aircraft  has  on 
any  sensed  track  is  organic  because  the  network  does  not  do 
any  refinement  of  the  sensed  track,  and  probability  is  high 
that  the  track  will  not  be  able  to  be  retrieved  over  the  voice- 
only  network. 

b.  For  Link  16,  best  source  of  information  each  aircraft  has  on 
any  sensed  track  is  organic  because  the  network  does  not  do 
any  refinement  of  the  sensed  track. 

2.  Compute  the  probability  that  the  Blue  aircraft  has  knowledge  of 
each  unique  track  within  a  latency  band,  by  computing  the 
probability  that  the  aircraft  has  information  on  at  least  one  of 
the  corresponding  sensed  tracks  within  the  latency  band  (either 
organic  or  retrieved  from  the  network). 

a.  For  voice  only,  this  gives  rise  to  an  array  of  probabilities.  Note 
that  tracks  known  within  the  less-than-one-second  band  result 
solely  from  organic  sensing. 

b.  For  Link  16,  this  gives  rise  to  an  array  of  probabilities  that 
tend  to  be  much  greater  than  for  voice  only.  Notably,  if  one 
Blue  fighter  detects  an  aircraft  (including  itself)  within  the 
less-than-one-second  band,  all  other  Blue  aircraft  share  the 
same  knowledge. 
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Phase  2:  Calculate  the  measurements  for  each  of  the  four  quality 
of  individual  information  metrics. 

3.  For  completeness  of  detection  and  correctness  of  ID,  the  mea¬ 
surement  equals  the  maximum  probability  of  “knowing”  a 
unique  air  track  from  either  the  less-than-one-second  or  one-  to 
ten-second  latency  bands.  This  stems  from  track  detection  and 
ID  being  insensitive  to  track  latency  (see  results  below) . 

4.  For  correctness  of  location  and  correctness  of  velocity,  recall  that 
tracks  known  within  the  less-than-one-second  band  have  a 
weight  of  1.0,  and  tracks  known  within  the  one-  to  ten-second 
band  have  weight  0.25.  Then,  to  compute  these  metrics,  select 
the  maximum  weighted  probability  of  “knowing”  the  air  track 
from  either  of  the  two  latency  bands  (see  results  below) . 


Individual  Platform  Results 

Figure  7.2  compares  quality  of  shared  information  scores  for  each 
Blue  aircraft  in  both  the  voice-only  and  Link  16-equipped  MCPs 
along  four  metrics  (all  normalized  between  0  and  1).  The  voice-only 
positions  are  similar  to  the  initial  organic  positions,  with  a  slight 
boost  resulting  from  tracks  reported  over  the  voice  channel.  The  Link 
16  positions  equal  the  maximum  of  the  initial  organic  positions.  The 
only  planes  not  known  to  a  near-real  time  precision  by  all  Blue  air¬ 
craft  are  Red  3  and  Red  4  (which  are  sensed  only  by  AWACS) . 

Figure  7.2  summarizes  and  compares  the  individual  information 
position  for  each  Blue  aircraft  across  each  of  the  four  major  metrics, 
as  follows: 

•  Completeness:  Detection  is  the  expected  percentage  of  all  air 
tracks  in  the  engagement  (four  Red  and  five  Blue  aircraft)  that 
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Figure  7.2 

Comparing  Quality  of  Individual  Information  Across  MCPs 
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the  Blue  aircraft  either  detects  directly  or  has  reported  to  it  in 
the  last  ten  seconds; 

•  Correctness:  Identification  is  the  expected  percentage  of  the  air 
tracks  for  which  the  Blue  aircraft  has  correct  combat  ID — i.e., 
Red,  Blue,  or  neutral/civilian  aircraft.  If  the  air  track  ID  is  cor¬ 
rect  a  score  of  1.0  is  given.  If  it  is  incorrect  or  designated  as 
unknown,  a  score  of  zero  is  given. 

•  Correctness:  Location  is  the  expected  percentage  of  air  tracks  for 
which  the  Blue  aircraft  has  a  location  report  (either  from  direct 
detection  or  network  communications  links).  If  the  location 
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report  is  less  than  one  second  old,  it  is  considered  to  be  “near 
real  time,”  allowing  for  precise  maneuvering  and  cuing  fire 
control  systems,  and  has  a  value  of  1.0.  If  the  report  is  one  to  ten 
seconds  old,  it  is  considered  to  be  “non-real  time,”  suitable  only 
for  general  cuing,  and  has  a  value  of  0.25.  If  the  report  is  more 
than  10  seconds  old,  it  is  given  a  value  of  zero,  as  described  ear¬ 
lier. 

•  Correctness:  Velocity  is  the  expected  percentage  of  air  tracks  for 
which  the  aircraft  has  a  velocity  report.  As  with  Correctness: 
Location,  the  velocity  report  has  a  value  of  1.0  if  it  is  less  than 
one  second  old,  0.25  if  it  is  one  to  ten  seconds  old,  and  zero  if  it 
is  older  than  that. 

As  shown,  Blue  aircraft  in  the  Link  16-equipped  MCP  had 
much  higher  quality  of  information  scores  than  the  voice-only  MCP, 
especially  for  the  location  and  velocity  metrics  (which  rely  heavily  on 
precise,  real-time  air  track  updates). 


Overall  Results 

Averaging  across  all  planes  and  all  track  quality  metrics: 

•  The  voice-only  MCP  has  an  overall  quality  of  individual  infor¬ 
mation  score  of  0.4. 

•  The  Link  16  MCP  has  an  overall  score  of  0.91. 

Both  of  these  scores  were  generated  from  the  exact  same  quan¬ 
tity  of  organic  information  (average  score  of  0.28).  The  Link  16  net¬ 
work  multiplies  the  utility  of  sensed  information  by  allowing  it  to  be 
shared  across  the  entire  Blue  force.  Figure  7.3  compares  these  two 


scores. 
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Figure  7.3 

Overall  MCP  Scores  for  Quality  of  Individual  Information 


CHAPTER  EIGHT 

Degree  of  Shared  Information 


Specific  Metrics,  Inputs,  and  Calculations 

The  subsidiary  attributes  that  constitute  the  degree  of  shared  infor¬ 
mation  are  largely  identical  to  those  for  quality  of  individual  informa¬ 
tion,  and  their  metrics  are  largely  calculated  the  same  way.  The 
primary  difference  is  that  the  metrics  are  computed  for  sets  of  infor¬ 
mation  shared  across  multiple  individuals  rather  than  computed  for 
the  information  held  by  each  individual.  There  is  one  additional  attri¬ 
bute:  the  extent  of  sharing,  which  describes  the  information  elements 
actually  held  in  common  across  different  force  members.  The  attri¬ 
bute  has  two  metrics:  the  proportion  of  information  in  common 
across  force  entities  and  the  proportion  of  force  entities  that  share  an 
information  element. 

In  our  analysis  of  the  degree  of  shared  information,  we  focused 
on  analogues  of  the  information  quality  metrics,  now  applied  to 
shared  rather  than  individual  track  information  (completeness  of 
track  detections,  correctness  of  track  identifications,  correctness  of 
location,  and  correctness  of  velocity).  The  new  metrics  have  the  same 
inputs  and  are  calculated  in  a  manner  similar  to  those  for  quality  of 
individual  information,  with  a  few  changes: 

•  Only  information  transmitted  across  the  networks  is  considered 
(i.e.,  organically  known  information  is  excluded  from  consider¬ 
ation).  Thus,  the  new  metric  provides  the  weighted  probability 
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that  a  given  Blue  aircraft  has  retrieved  information  on  a  given 
unique  track  over  the  network  (whether  voice-only  or  Link  1 6) . 

•  The  metrics  are  calculated  with  respect  to  sharing  the  “available” 
track  information  across  the  Blue  force.  Therefore,  the  scores  are 
not  penalized  for  failing  to  share  track  information  that  is 
unknown  organically.  For  example,  in  the  case  study,  two  Red 
aircraft  tracks  (Red  3  and  Red  4)  are  not  detected  in  near  real 
time  (in  the  less-than-one-second  latency  band)  by  any  Blue  air¬ 
craft  and  so  are  excluded  from  the  measurement  calculations. 


Individual  Platform  Results 

Figure  8.1  shows  the  quality  of  the  available  track  information  shared 
across  the  Blue  aircraft.  The  charts  in  Figure  8.1  apply  equally  to 

Figure  8.1 

Comparing  Degree  of  Shared  Information  Across  MCPs 
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every  Blue  aircraft  in  the  force  package  because  all  of  them  are 
assumed  to  be  listening  to  the  same  voice  channel  and/or  receiving 
the  same  Link  16  broadcasts.  In  general,  we  see  that  the  voice-only 
system  manages  to  share  around  25  percent  (Red)  and  20  percent 
(Blue)  of  the  detection  and  ID  information,  and  12  percent  (Red) 
and  6  percent  (Blue)  of  the  location  and  velocity  information  (taking 
into  account  the  degradation  involved  in  transmitting  near-real  time 
tracks  over  voice),  whereas  the  Link  16  system  shares  all  available 
information. 


Overall  Results 

Averaging  across  all  measurements: 

•  The  voice-only  MCP  scores  0.15  for  degree  of  shared  informa¬ 
tion. 

•  The  Link  16  MCP  scores  1.0  because  every  available  air  track 
that  could  be  shared  was  shared  to  the  highest  degree  of  accuracy 
in  this  case  study. 

Figure  8.2  compares  the  two  MCP  scores  for  quality  of  shared 
information. 

These  results  clarify  what  was  implied  by  the  quality  of  individ¬ 
ual  information  results — that  the  voice-only  network  provides  only  a 
marginal  contribution  to  the  Blue  aircraft’s  information  positions, 
whereas  Link  16  largely  ensures  that  what  is  known  organically  by 
one  aircraft  is  known  by  all  Blue  aircraft. 
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Figure  8.2 

Overall  MCP  Scores  for  Quality  of  Shared  Information 
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CHAPTER  NINE 

Quality  of  Individual  Sense-Making:  Awareness 


Major  Factors  Contributing  to  Individual  Sense-Making 

A  number  of  major  factors  go  into  individual  sense-making  at  the 
awareness,  understanding,  and  decision  levels,  such  as  doctrinal 
impacts,  organizational  impacts,  training,  experience,  and  the  quality 
of  the  interactions  between  force  members.  Figure  9.1  summarizes  a 
number  of  these  factors  contributing  just  to  individual  awareness, 
showing  which  ones  relate  directly  to  the  quality  of  information 
measurements,  which  relate  to  the  NCO  CF  quality  of  interactions 
attributes  (not  included  in  this  study)  and  are  exogenous. 

Understanding  of  these  cognitive  processes  in  the  context  of  a 
military  operation  is  limited  because  research  into  them  is  very  recent. 
Unlike  the  previous  concepts,  which  can  be  assessed  through  precise 
observation  and  assessment,  the  sense-making  concepts  must  be 
assessed  indirectly  because  it  is  impossible  to  view  someone’s  knowl¬ 
edge  and  thought  processes.  Further,  sense-making  is  known  to 
depend  on  many  factors  that  are  not  well  understood  and  are  assessed 
qualitatively  and  whose  impacts  on  sense-making  are  even  less  well 
understood. 

In  this  report,  then,  we  rely  on  approximate  models  of  aware¬ 
ness,  understanding,  and  decisions.  Awareness  closely  relates  to  a  per¬ 
son’s  mental  incorporation  of  the  information  they  receive,  so  we 
model  awareness  as  estimating  how  well  the  pilots  know  the  aircraft 
track  information  they  receive. 
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Figure  9.1 

Major  Factors  Contributing  to  Individual  Sense-Making 


Exogenous  Quality  of  interactions 

Quality  of  individual  information  inputs  (not  included  in  case  study) 


Understanding  is  more  complicated,  relating  to  how  well  a  per¬ 
son  can  build  on  this  awareness,  determining  what  events  will  likely 
happen  in  the  future  and  how  they  might  be  addressed  (generating 
options  and  likely  outcomes).  Thus,  in  this  report,  understanding  is 
modeled  by  estimating  whether  a  person  knows  enough  to  make 
good  decisions  or  at  least  has  the  potential  to  make  good  decisions. 
Therefore,  we  do  not  address  understanding  as  a  separate  concept  but 
rather  address  it  in  conjunction  with  decisionmaking  later  in  this 
report. 

Note  that  from  the  pilot  interviews,  we  have  made  some  useful 
observations  on  awareness  and  understanding  from  a  qualitative  per¬ 
spective.  With  the  voice-only  network,  most  of  the  voice  channel  is 


Quality  of  Individual  Sense-Making:  Awareness  59 


dedicated  strictly  to  information  sharing,  which  leaves  little  time  for 
pilots  to  build  more  detailed  understanding  of  the  battlespace  or  to 
collaborate  with  others  to  assist  in  building  such  detailed  under¬ 
standing.  In  addition,  the  fact  that  the  track  information  is  auditory 
and  on  a  time  delay  leaves  pilots  to  create  an  approximate  mental  pic¬ 
ture  of  each  tracked  plane’s  position.  In  contrast,  with  the  Link  16 
network,  aircraft  track  information  is  immediately  broadcast  over  the 
force  visually,  so  pilots  do  not  have  to  construct  a  time-delayed  men¬ 
tal  image — the  Link  16  display  does  it  for  them.  This  leaves  more 
time  for  pilots  to  gain  a  deeper  understanding  of  the  state  of  the  bat¬ 
tlespace  and  what  to  do  about  it.  In  addition,  they  also  have  unfet¬ 
tered  access  to  the  voice  channel  if  they  want  to  discuss  what  to  do 
with  the  information  or  issue  related  commands. 


Specific  Metrics,  Inputs,  and  Calculations 

For  this  case  study,  we  use  a  simple  approximation  to  estimate  quality 
of  individual  awareness  metrics  directly  from  the  quality  of  informa¬ 
tion  metrics  (completeness  of  tracks;  correctness  of  track  ID,  location, 
and  velocity) . 

As  with  the  quality  of  information  metrics,  we  want  to  know 
whether  a  Blue  aircraft  pilot’s  awareness  of  the  aircraft  tracks  is  in 
near  real  time  (mental  image  of  aircraft  track  is  within  one  second  of 
its  real  position  and  velocity),  or  cue-only  (mental  image  is  within 
one  to  ten  seconds  of  its  real  position  and  velocity),  with  a  much 
greater  weight  placed  on  the  former  than  the  latter  (1.0  versus  0.25 
for  the  position  and  velocity  metrics).  Rather  than  simply  copy  the 
individual  information  metrics,  however,  we  introduce  the  idea  of 
“memory.”  For  the  track  information  in  the  one-  to  ten-second  band, 
we  assume  a  50  percent  chance  that  the  information  will  be  valid  in 
the  next  ten-second  window,  on  the  grounds  that  the  pilot  may 
remember  the  information  accurately  enough  and  that  the  tracked 
plane  has  not  deviated  from  its  path  far  enough  to  make  the  pilot’s 
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mental  image  of  it  invalid.  Track  information  in  the  less-than-one- 
second  band  must  be  refreshed  every  second  to  remain  valid. 


Individual  Platform  Results 

Figure  9.2  displays  the  resulting  individual  awareness  scores  for  each 
Blue  pilot.  Note  that  the  scores  for  the  Link  16  MCP  are  exactly  the 
same  as  for  individual  information  because  all  track  information  is 
updated  as  frequently  as  it  is  obtained.  That  is,  information  obtained 


Figure  9.2 

Comparing  Quality  of  Individual  Awareness  Across  MCPs 
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in  less  than  one  second  is  distributed  for  the  pilots  multiple  times  per 
second  and  information  from  the  AWACS  is  updated  every  ten  sec¬ 
onds,  as  it  is  obtained.  The  main  differences  in  scores  are  for  the 
voice-only  MCP,  where  “memory”  allows  pilots  to  employ  track 
information  reported  more  than  ten  seconds  ago. 


Overall  Results 

Averaging  across  all  quality  of  individual  awareness  scores  for  all  Blue 

aircraft  and  tracks: 

•  The  voice-only  MCP  pilots  have  an  aggregate  individual  aware¬ 
ness  score  of  0.45,  which  is  a  slight  improvement  over  rhe  aggre¬ 
gate  individual  information  score  of  0.40. 

•  The  Link  16  MCP  pilots  have  an  aggregate  individual  awareness 
score  of  0.91,  which  equals  the  individual  information  score. 
Figure  9.3  compares  the  two  MCP  scores. 

Figure  9.3 

Overall  MCP  Scores  for  Quality  of  Individual  Awareness 
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CHAPTER  TEN 

Quality  of  Individual  Sense-Making: 
Decisionmaking 


Major  Factors  Contributing  to  Individual  Decisionmaking 

We  now  consider  the  quality  of  individual  decisionmaking  (and  indi¬ 
vidual  understanding  leading  to  decisionmaking).  As  with  other 
aspects  of  individual  sense-making,  individual  decisionmaking  is 
complicated  and  not  well  understood  because  it  takes  place  almost 
entirely  in  the  cognitive  realm.  Figure  10.1  summarizes  factors  con¬ 
tributing  to  the  decisionmaking  measurements  and  shows  whether 
these  factors  are  derived  from  the  quality  of  individual  awareness 
scores,  the  NCO  CF  quality  of  interactions  attributes,  or  other, 
exogenous  factors.  As  shown  on  the  figure,  we  do  not  model  individ¬ 
ual  understanding  directly  in  this  case  study,  so  its  scores  are  implic¬ 
itly  included  in  the  calculation  of  individual  decisionmaking  scores. 
Figure  10.1  also  shows  the  three  attributes  for  individual  decision¬ 
making  that  are  comparable  to  the  completeness,  correctness,  and 
accuracy  measures  used  to  date — completeness  of  the  decisions 
(whether  the  decisions  encompass  necessary  depth,  breadth,  and  time 
horizons),  appropriateness  (whether  the  decisions  are  consistent  with 
existing  understanding,  command  and  intent  and  values — analogous 
to  correctness),  and  accuracy  (whether  the  decisions  are  sufficiently 
precise  for  a  particular  use).  Note,  however,  that  we  do  not  use  these 
three  attributes  to  model  individual  decisionmaking  in  this  case 
study. 
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Figure  10.1 

Factors  Contributing  to  Quality  of  Individual  Decisionmaking 
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Interviews  with  experienced  pilots  revealed  that  the  improved 
quality  of  information  under  Link  16  improved  situational  awareness 
and  subsequent  decisionmaking  in  two  ways.  First,  in  general,  the 
pilots  with  access  to  the  Link  16  network  reported  spending  less  time 
building  situational  awareness  (i.e.,  determining  where  the  Red  and 
Blue  aircraft  are)  than  did  pilots  with  access  limited  to  the  voice-only 
network.  In  the  voice-only  network,  pilots  had  to  continually  listen  to 
voice  traffic  describing  air  tracks,  mentally  convert  each  description 
into  a  velocity  and  location,  predict  where  the  aircraft  would  likely  be 
based  on  the  last  report,  and  perform  these  mental  calculations  while 
listening  to  further  incoming  reports.  This  process  of  gaining  aware¬ 
ness  was  described  as  slow  (restricted  by  voice  transmissions),  men¬ 
tally  taxing,  and  potentially  error-prone.  Further,  because  the  manual 
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mental  process  of  building  awareness  is  error-prone  under  the  stress¬ 
ing  conditions  of  combat,  it  is  likely  that  pilot  situational  awareness 
information  is  not  entirely  common  across  the  MCP.  In  other  words, 
situational  awareness  is  shared  and  interpreted  imperfectly  among 
pilots  over  voice  channels. 

In  contrast,  in  the  Link  16  network,  pilots  are  presented  with  a 
continually  updated  image  visually  displaying  the  precise  positions 
and  velocities  of  all  detected  aircraft  in  the  battlespace;  the  resulting 
process  of  gaining  situational  awareness  was  much  faster,  almost 
automatic  (no  mental  calculations  required),  and  accurate.  The 
resulting  time  compression  in  obtaining  information  and  awareness 
with  Link  16  is  shown  in  Figure  10.2.  This  freed  time  could  be  used 
to  both  spend  more  time  considering  alternative  courses  of  action, 
which  will  tend  to  lead  to  better  decisions,  and  make  more  decisions 
in  a  given  period  of  time,  which  (assuming  the  decisions  are  reason¬ 
able)  should  lead  to  more  targets  destroyed.  Notably,  the  freed  time 
also  allows  the  wingman  time  for  sense-making  and  making  decisions 
to  engage  targets,  as  opposed  to  spending  virtually  all  his  time  gath¬ 
ering  and  monitoring  critical  information,  as  in  conventional  doc¬ 
trine.  This  is  depicted  graphically  in  Figure  10.2.  It  is  important  to 
note  that  the  time  line  in  the  figure  is  qualitative  in  nature.  The  time 
needed  by  pilots  to  build  their  situational  awareness  could  not  be 
quantified  in  the  case  study  even  though  the  differences  in  this  attri¬ 
bute  were  thought  to  be  very  significant  according  to  pilots  who  had 
experience  with  Link  16. 

The  second  influence  factor  is  that  the  increased  knowledge  of 
both  Red  and  Blue  locations  enables  new  tactics  to  be  employed.  We 
use  this  second  influence  factor,  which  builds  on  the  first  discovered 
in  our  analysis  (and  as  depicted  in  Figure  10.1),  and  discuss  its  limita¬ 
tions  after  discussing  the  approach. 

In  conventional  Air  Force  doctrine  (with  a  voice-only  network), 
half  of  the  Blue  fighters — the  wingmen — are  put  on  defensive  patrol 
duty  while  the  flight  leads  engage  targets.  Furthermore,  the  Blue  air- 
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Figure  10.2 
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craft  have  limited  awareness  of  the  locations  of  the  other  Blue  forces 
and  so  cannot  act  cooperatively. 

From  discussions  with  pilots  who  participated  in  the  JTIDS 
Special  Operations  Project,  we  have  identified  four  types  of  air-to-air 
combat  tactics  that  were  executed  more  effectively  by  the  fighter 
pilots.  This  improvement  in  tactics  execution  was  enabled  by  the 
increased  information  and  decreased  time  required  for  information¬ 
gathering  in  Link  1 6-equipped  force  packages,  as  discussed  on  the  last 
figure.  The  pilots  were  able  to  improve  execution  of  air  combat  tac¬ 
tics  by  taking  advantage  of  their  increased  awareness,  as  well  as  the 
increased  time  they  had  available  for  decisionmaking.  These 
improved  tactics  are  shown  in  Figure  10.3. 
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Figure  10.3 

Improved  Air-to-Air  Tactics  Execution  Enabled  by  Improved  Awareness 
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The  first  of  these  is  simply  an  increased  number  of  engagements 
in  the  same  period.  This  tactic  is  possible  because  pilots  with  Link  16 
can  quickly  recognize  the  most  efficient  attack  trajectories.  This  is  an 
important  consideration  because  (according  to  the  pilots)  the  fighters 
only  have  a  limited  time  to  engage  before  they  run  low  on  fuel  and 
must  return  to  base. 

The  second  tactic  is  the  employment  of  the  wingman  as  a 
combatant  rather  than  a  defensive  patroller.  With  Link  16,  all  threats 
are  known  by  all  Blue  aircraft.  The  flight  lead  has  more  options  for 
employing  the  wingman  as  a  primary  shooter  because  of  the  higher 
levels  of  individual  and  shared  understanding  of  the  engagement, 
effectively  doubling  the  firepower  of  the  fighter  package. 

The  third  advanced  tactic  is  the  use  of  other  planes’  track  infor¬ 
mation  to  perform  earlier  and  more  accurate  vectoring,  which  allows 
a  Blue  fighter  to  enter  an  engagement  from  a  position  of  maximum 
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advantage,  before  the  Blue  fighter’s  radar  (or  the  Red  plane’s  radar, 
for  that  matter)  can  detect  the  engaging  plane. 

The  fourth  tactic  is  the  use  of  “ambush  CAP”  tactics  and  the  use 
of  terrain  to  trap  and  destroy  Red  aircraft.  Because  all  Blue  locations 
are  known  by  Blue  aircraft — even  if  those  aircraft  are  operating  in 
radar  or  IFFN  silence — they  have  more  options  to  engage  targets. 
One  example  of  “ambush  CAP”  tactics  is  when  a  Blue  fighter  chases  a 
Red  fighter  toward  other  Blue  aircraft.  The  latter  Blue  aircraft  have 
their  radars  turned  off  (or  are  hiding  in  a  canyon)  so  the  Red  fighter 
is  not  likely  to  know  that  the  latter  Blue  aircraft  are  present.  Then, 
when  the  Red  aircraft  is  chased  into  range,  the  other  Blue  fighters  will 
suddenly  engage  the  Red  fighter,  surprising  it  and  likely  destroying  it 
with  minimal  damage  to  Blue  aircraft. 


Specific  Metrics,  Inputs,  and  Calculations 

For  this  case  study,  the  metric  representing  quality  of  individual  deci¬ 
sionmaking  is  the  probability  that  the  Blue  aircraft  have  an  awareness 
of  the  battlespace  sufficient  to  execute  each  of  the  advanced  tactics 
described  above.  Note  that  many  exogenous  variables  contribute  to 
whether  the  Blue  fighters  can  decide  to,  and  correctly  execute,  the 
types  of  advanced  tactics  mentioned  in  the  previous  figure,  ranging 
from  individual  training  to  the  team  hardness  of  the  force  package  to 
whether  Air  Force  C2  allows  the  advanced  tactics.  H owever,  because 
this  report’s  purpose  is  to  compare  the  impacts  of  the  Link  16  and 
voice-only  architectures,  we  focus  strictly  on  whether  the  quality  of 
individual  awareness  (and  hence  the  quality  of  individual  informa¬ 
tion)  is  sufficient  to  allow  the  tactics  to  be  employed.  Table  10.1  pre¬ 
sents  a  table  of  awareness  requirements  for  each  advanced  tactic.  Note 
that  the  table  makes  reference  to  an  “immediate  battle  area.”  This  is 
the  space  surrounding  each  Blue  aircraft  in  which  the  fighter  can 
actually  engage  another  plane  or  be  engaged  by  another  plane;  it  is  a 
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Table  10.1 

Sense-Making  Requirements  to  Employ  Improved  Tactics  Execution 
(Notional) 


Tactic 

Near-Real  Time  Awareness  (less- 
than-one-second  band) 

Cue-Only  Awareness 
(one-  to  ten-second  band) 

Accelerated 

engagements 

All  aircraft  in  immediate  battle 
area  (to  enable  rapid  target 
selection) 

All  aircraft  in  battlespace 
(eliminates  the  need  to  wait 
for  cuing  info  and  assists 
determination  of 
engagement  priorities) 

Combatant 

wingman 

All  aircraft  in  immediate  battle 
area,  plus  other  Blue  aircraft  in 
flight  (to  meet  protection  role) 

All  Red  aircraft  in  battlespace 
(prevents  surprise) 

Earlier  and  more 
accurate  vector¬ 
ing 

All  aircraft  in  immediate  battle 

area 

All  aircraft  in  battlespace 
(provides  info  needed  to 
vector  on  Red  aircraft  outside 
immediate  battle  area) 

"Ambush  CAPs" 

All  Blue  aircraft  in  battlespace 
(needed  to  create  precise  forma¬ 
tions);  all  Red  aircraft  in  immedi¬ 
ate  battle  area 

All  Red  aircraft  in  battlespace 
(provides  general  cuing  info 
for  formations  and  avoidance 
of  other  aircraft) 

small  subset  of  the  whole  battlespace,  which  can  be  hundreds  of  miles 
in  diameter  (equivalent  to  the  radar  range  of  the  AWACS  aircraft) . 


Results 

Figure  10.4  compares  the  requirements  for  improved  tactics  execu¬ 
tion  to  the  state  of  the  battlespace  (in  terms  of  which  planes  are  in 
which  battle  areas) ,  and  the  Blue  fighters’  quality  of  individual  aware¬ 
ness.  The  quality  of  situational  awareness  within  the  Link  1 6  network 
is  sufficient  to  enable  the  improved  tactics,  whereas  the  situational 
awareness  within  the  voice-only  network  is  insufficient  to  use  these 
same  tactics  as  effectively. 

Weighting  the  potential  to  perform  each  tactic  equally  yields: 

•  The  voice-only  MCP  has  an  overall  quality  of  individual  deci¬ 
sionmaking  score  of  0.2  because  this  MCP  can  perform  the  four 
tactics  considered  with  only  low  to  moderate  effectiveness. 
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Figure  10.4 

Sense-Making  Requirements  Met  by  the  MCPs 
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•  The  Link  16  MCP  has  an  overall  score  of  1.0  because  this  MCP 
showed  significant  improvement  in  executing  all  four  tactics. 

Figure  10.5  compares  the  two  MCP  scores. 

Note  that  the  values  presented  in  Figures  10.4  and  10.5  are  not  based 
on  data  about  the  actual  number  or  quality  of  the  decisions  made  the 
pilots.  They  are  simply  based  on  Link  16’s  ability  to  effectively  sup¬ 
port  all  four  types  of  tactics  in  Figure  10.4.  We  have  chosen  this 
approach  because  we  do  not  have  any  data  on  the  decisions  made  and 
because  we  believe  that  decisions  embodied  in  the  improved  tactics 
execution  generally  are  superior  decisions  (with  the  exact  decisions 
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appropriate  to  make  depending  on  the  particulars  of  the  engage¬ 
ment)  . 

One  limitation  of  this  approach  is  that  the  values  are  based  on 
subjective  data  determined  through  interviews  with  Link  16  pilots 
and  not  on  objective  measures  of  improved  decisionmaking  or  the 
quality  of  decisions.  The  rankings  of  the  two  MCPs  considered  are 
therefore  only  relative  and  not  absolute.  Put  another  way,  the  rank¬ 
ings  reflect  only  the  perceived  improvement  in  tactics  execution.  This 
is  as  much  a  measurement  of  agility  as  of  the  quality  of  the  decisions 
(see  Chapter  Eleven).  The  precision  of  the  numbers  reported  has  been 
reduced  to  reflect  the  low-level  of  resolution  that  this  analysis  allows. 


Linking  Decisionmaking  to  Mission  Effectiveness 

The  last  step  in  applying  the  NCO  CF  is  to  link  the  decisionmaking 
scores  to  decision  and  action  synchronization  and  hence  to  mission 

Figure  10.5 

Overall  MCP  Scores  for  Quality  of  Individual  Decisionmaking 
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effectiveness.  Table  10.2  repeats  the  table  of  average  loss  exchange 
ratios  from  the  JTIDS  study.  As  shown,  Link  16  increased  the  aver¬ 
age  force  effectiveness  by  1 60  percent. 

We  use  this  table  to  calculate  overall  MCP  scores  for  mission 
effectiveness.  Because  we  do  not  have  sufficient  data  to  define  “per¬ 
fect”  kill  ratios,  we  define  the  maximum  observed  kill  ratios  (in  this 
case,  for  Link  16)  to  be  1.0,  and  score  the  voice-only  MCP’s  mission 
effectiveness  as  its  average  kill  ratios  divided  by  the  maximum 
observed  kill  ratios.  Doing  so,  we  find: 

•  By  definition,  the  Link  16  MCP  scores  1.0  for  mission  effec¬ 
tiveness 

•  The  voice-only  MCP  scores  only  0.38  for  mission  effectiveness. 

Figure  10.6  compares  these  two  scores  graphically. 

The  natural  way  to  model  the  link  between  individual  decisions 
and  effectiveness  would  be  to  estimate  the  marginal  impact  of  the 
ability  to  employ  each  of  the  advanced  tactics  previously  discussed. 
However,  we  do  not  have  enough  data  to  attempt  to  compute  the 
marginal  impacts  in  this  report. 


Table  10.2 

Results  of  the  JTIDS  Operational  Special  Project 
(Average  Red-to-Blue  Loss  Exchange  Ratios) 


Kill  Ratio 

Voice  Only 

Voice  Plus  Link  16 

(MCP  1) 

(MCP  2) 

Day 

3.10:1 

8.11:1 

Night 

3.62:1 

9.40:1 
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Figure  10.6 

Overall  MCP  Scores  for  Mission  Effectiveness 
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CHAPTER  ELEVEN 

Conclusions 


In  this  chapter,  we  discuss  the  results  from  the  analysis  conducted  in 
the  air-to-air  combat  case  study  and  the  application  of  the  NCO  CF. 


Summary 

We  can  summarize  the  results  of  this  case  study  by  saying  that  the 
overall  conclusions  of  the  NCW  hypothesis  have  been  validated  in 
this  examination  of  the  air-to-air  combat  mission.  Despite  starting 
with  similar  airframes,  training,  doctrine,  and  organic  sensing  capa¬ 
bilities,  the  Link  1 6-equipped  MCP  was  able  to  take  advantage  of  the 
information  shared  within  the  MCP  through  Link  16  and  voice  net¬ 
works  far  more  effectively  than  the  voice-only  MCP.  Because  of  this, 
the  quality  of  information  available  to  individual  fighter  pilots  was 
increased  significantly.  They  did  not  have  to  spend  as  much  time 
building  their  own  picture  of  the  battlespace  or  their  own  awareness 
of  the  battlespace.  Instead,  they  have  more  time  available  for  deci¬ 
sionmaking.  While  not  proved  conclusively  in  this  case  study,  we 
believe  on  the  basis  of  interviews  with  experienced  pilots  who  have 
flown  with  Link  16  that  they  were  on  average  able  to  make  better 
decisions  and  make  decisions  earlier  in  the  opening  gambits  of  tactical 
air-to-air  engagements.  This  resulted  in  greatly  increased  force  effec¬ 
tiveness  as  measured  in  the  landmark  JTIDS  experiments  that  were 
conducted  several  years  ago. 
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To  graphically  show  the  improvements,  Figure  11.1  traces  the 
average  scores  of  the  Link  16  and  voice-only  MCPs  all  the  way 
through  the  NCO  CF,  starting  from  quality  of  organic  information 
and  ending  with  improved  mission  effectiveness. 

As  hypothesized  by  the  NCW  tenets,  the  robustly  networked 
force  enabled  by  Link  16  improved  information  sharing  and  the 
resulting  quality  of  information,  which  enhanced  shared  situational 
awareness,  which  in  turn  enabled  self-synchronization  (in  this  case 
study,  as  measured  by  the  ability  to  make  improved  decisions  and  by 
improved  tactics  execution)  and  which  resulted  in  dramatically 
increased  mission  effectiveness  as  measured  by  the  kill  ratios. 

Figure  11.1 

Complete  Comparison  of  MCPs  Using  Summary  Metrics 
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The  air-to-air  combat  case  study  provided  quantitative  data  con¬ 
cerning  the  benefits  of  Link  16  to  both  information  distribution  and 
to  lower  levels  of  individual  awareness.  Furthermore,  it  showed  that 
these  increases  in  situational  awareness  likely  enabled  the  improved 
execution  of  the  four  advanced  tactics  described  in  the  previous 
chapter,  which  in  turn  led  to  the  observed  increases  in  kill  rates. 


Lessons  Learned  from  Applying  the  NCO  CF  to  the 
Air-to-Air  Case  Study 

Below,  we  describe  the  lessons  learned  about  the  NCO  CF  from  the 
air-to-air  combat  case  study  and  the  limitations  of  the  study  as  an 
exploration  of  the  NCO  CF. 

Applicability  of  the  NCO  CF.  The  case  study  provided  valuable 
insights  into  the  application  of  the  NCO  CF  and  associated  metrics. 
In  particular,  we  have  shown  that  it  is  feasible  to  approximate  the 
chains  of  influences  through  the  NCO  CF  and  do  so  in  a  highly 
meaningful  manner.  For  example,  rather  than  saying,  “Link  16  was 
better  than  voice-only  because  its  numerical  scores  were  higher,”  it  is 
possible  to  say,  “Link  16  was  better  than  voice-only  because  its  data- 
handling,  bandwidth,  accuracy,  and  reach  provided  all  Blue  aircraft 
with  the  seven  real-time,  accurate  tracks  needed  to  improve  tactics 
execution,  which  in  turn  greatly  increased  kill  ratios  in  the  JTIDS 
experiments.” 

Furthermore,  this  case  study  showed  that  an  analysis  using  the 
NCO  CF  is  possible,  even  in  the  absence  of  detailed  information.  In 
this  instance,  we  had  only  knowledge  of  the  capabilities  employed 
and  the  kill  ratio  but  were  still  able  to  develop  a  credible  “story” 
about  how  improvements  in  the  degree  of  networking  led  to  an 
improved  kill  ratio. 

We  recommend  that  more  complex  case  studies  be  conducted  to 
further  develop  an  understanding  of  NCW  and  the  NCO  CF. 
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Limitations  of  the  NCO  CF.  In  performing  the  analysis,  we  used 
many  of  the  metrics  and  did  not  identify  any  major  deficiencies  at  the 
lower  levels  of  the  NCO  CF.  ITowever,  we  did  not  use  traditional 
interpretations  of  the  metrics  for  decision  quality  because  we  had  no 
data  on  individual  decisions.  Instead,  we  analyzed  the  advanced  tac¬ 
tics  reportedly  employed  in  air-to-air  combat  when  using  Link  16. 

This  analysis  could  be  interpreted  within  the  context  of  the 
NCO  CF  in  several  ways: 

•  Link  16  enabled  the  wingman  to  be  used  as  a  combatant,  thus 
resulting  in  a  more  agile  (flexible)  decisionmaking  process  and 
more  offensive  firepower 

•  Link  16  enabled  accelerated  engagements  and  thus  resulted  in 
increased  decision  timeliness 

•  Link  16  enabled  earlier  and  more  accurate  vectoring  and 
“ambush  CAP”  tactics  that  resulted  in  more  “appropriate”  deci¬ 
sions. 

ITowever,  the  flexibility  metrics  of  the  NCO  CF  do  not  reflect 
the  quality  of  the  options,  and  decisions  made  without  Link  16  could 
also  be  “appropriate” — particularly  given  the  available  information. 
Adding  other  metrics  that  reflect  the  “decisiveness”  or  “potential 
impact”  of  decisions  and  options  to  a  greater  degree  may  be  necessary. 

Another  potential  shortcoming  in  the  NCO  CF  is  that  the 
degree  of  information  share-ability  should  consider  not  only  the 
quantity  but  also  the  quality  of  information  posted  and  retrieved  and 
whether  the  information  is  correct,  incorrect,  or  irrelevant  to  the  mis¬ 
sion  under  consideration.  That  is,  it  should  consider  whether  the 
information  is  degraded  when  it  is  posted  and  retrieved.  While  it  did 
not  play  a  role  in  this  case  study,  it  should  be  recognized  that  these 
quantities  will  depend  on  the  quality  of  interactions  measurements 
and  whether  people  are  willing  to  post  and  retrieve  information. 

We  had  to  identify  numerous  relationships  between  the  metrics, 
the  underlying  data,  and  exogenous  variables.  ITowever,  this  was 
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expected,  given  the  generic  nature  of  the  framework,  and  we  did  not 
identify  any  relationships  that  should  clearly  be  added  to  the  top-level 
NCO  CF.  We  did  note  that  the  quantity  of  organic  information  (as 
opposed  to  its  quality)  affects  the  Degree  of  information  “share- 
ability”  because  more  information  leads  to  heavier  bandwidth 
demands.  Further,  because  we  did  not  model  interactions,  the  rela¬ 
tionship  between  individual  information  and  shared  information  was 
direct. 

The  exogenous  variables  needed  for  the  analysis  fall  into  two 
categories:  those  required  to  convert  information  about  the  force  into 
data  and  ultimately  metrics  to  evaluate  the  lowest-level  capabilities — 
organic  information  and  networking — and  those  required  to  evaluate 
the  higher-level  capabilities.  It  is  reasonable  to  assume  that  the  first 
set  do  not  need  to  be  captured  in  the  framework.  Other  exogenous 
variables  identified  that  may  need  to  be  explicitly  considered  for  later 
versions  of  the  framework  included: 

•  Training,  tactics,  procedures 

•  Size  of  the  network  and  quantity  of  organic  information 

•  Information  fusion  processes. 

Limitations  of  the  Air-to-Air  Case  Study.  While  the  air-to-air 
case  study  was  successfully  conducted,  it  did  not  stress  the  NCO  CF 
in  a  number  of  areas  that  should  be  addressed  in  future  studies. 

First,  the  air-to-air  scenario  is  relatively  simple.  It  contained  only 
nine  force  entities,  five  Blue  and  four  Red,  and  we  tracked  only  four 
pieces  of  relatively  simple  information:  detection,  identification 
(Blue/Red),  location,  and  velocity  as  well  as  their  relationships  to  four 
advanced  tactics.  Other  scenarios  with  more  force  elements  or  infor¬ 
mation  elements  would  stress  the  NCO  CF  more. 

For  example,  it  is  not  clear  from  this  analysis  if  the  NCO  CF 
would  satisfactorily  handle  such  information  elements  as  command 
intent  and  unit  capabilities  that  may  be  generated  from  using  large 
quantities  of  information. 


80  NCO  Case  Study:  Air-to-Air  Combat  With  and  Without  Link  16 


Second,  the  analysis  focused  mainly  on  the  easily  quantifiable 
capabilities — namely,  networking  and  information.  Tracking  the 
quality  of  interactions  and  cognitive  artifacts  is  much  more  complex. 
For  example,  shared  awareness  and  understanding  may  require  addi¬ 
tional  assessment  techniques  to  those  required  for  individual  aware¬ 
ness.  Similarly,  completing  the  chain  from  decisionmaking  to  mission 
effectiveness  will  require  collecting  data  on  the  marginal  impacts  of 
deciding  on  different  types  of  tactics  under  varying  conditions. 


Areas  for  Further  Research 

The  Air-to-Air  Case  Study  was  a  pilot  study,  and  as  such,  several  areas 
received  limited  treatment  and  would  benefit  from  added  attention. 

First  and  foremost  is  the  acquisition  of  data  describing  cognitive 
and  social  behavior.  The  current  modeling  of  quality  of  sense-making 
and  quality  of  Decisionmaking  is  highly  approximate  and  based  only 
on  a  small  number  of  informal  interviews. 

Second,  we  would  like  to  model  the  impact  of  nonmateriel 
changes  to  the  force  packages.  These  include  more  detailed  analysis  of 
the  changes  to  the  C2  concept,  tactics,  techniques,  and  procedures 
employed  in  the  experiments.  Similarly,  it  would  be  useful  to  model 
the  impact  of  changes  to  the  force  mix  beyond  Link  16  versus  voice- 
only,  such  as  changes  to  the  aircraft,  sensors,  and  weapons  involved  as 
well  as  individual  and  organizational  characteristics,  such  as  team 
hardness.  Together,  modeling  these  sets  of  changes  will  help  distin¬ 
guish  the  impact  of  the  network  on  mission  effectiveness  from  the 
impact  of  other  nonmaterial  and  force  mix  changes. 

Third,  we  would  like  to  expand  the  scenario  used  to  compute 
the  metrics  beyond  a  single  four-on-four  engagement.  This  will  mean 
both  calculating  metrics  over  a  wide  range  of  possible  engagements, 
varying  mission  scales,  force  balances,  and  information  positions 
(perhaps  using  Monte  Carlo  simulation)  and  having  some  ability  to 
account  for  dynamics  in  information  positions  over  time. 
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Finally,  we  would  like  to  use  the  same  overall  approach  in  this 
case  study  to  other  combat  areas.  We  believe  that  the  detailed 
approach  of  identifying  what  information  elements  can  be  generated 
and  shared  through  the  use  of  different  networks,  what  the  subse¬ 
quent  situational  awareness  is  across  the  joint  force,  and  what  types  of 
“advanced”  understanding  and  tactics  are  enabled  as  a  result  is 
broadly  applicable.  For  example,  the  approach  might  be  applied  in 
disparate  areas  ranging  from  joint  close  air  support  to  ground  maneu¬ 
ver.  Of  particular  interest  would  be  an  attempt  to  apply  the  approach 
to  stability  operations  and  missions  other  than  war — environments 
that  do  not  have  the  same  clarity  as  traditional  air-to-air  combat. 


APPENDIX 

Analytica  Model  for  the  Air-to-Air  Combat 
Example 


Model  Fundamentals 

This  appendix  describes  the  Analytica  model  used  to  calculate  the 
metrics  in  this  report.  Analytica  uses  a  multidimensional-array  nota¬ 
tion  and  structure  that  is  somewhat  different  from  common  mathe¬ 
matical  and  simulation  modeling  techniques,  so  we  begin  the 
appendix  with  a  discussion  of  model  fundamentals. 

At  the  top  level,  Analytica  models  comprise  a  number  of  nodes 
and  modules  linked  into  an  influence  diagram.  Each  node  in  the  dia¬ 
gram  either  calculates  a  portion  of  a  metric  or  contains  input  parame¬ 
ters  used  in  other  nodes’  calculations.  A  calculation  normally  depends 
on  the  output  of  other  nodes;  node  A  is  dependent  on  node  B  if  node 
A  uses  the  output  of  node  B  to  compute  a  metric.  Analytica  would 
show  the  former  dependency  with  an  arrow  (link)  from  node  B  to 
node  A.  Modules  consolidate  a  group  of  linked  nodes  into  what 
appears  as  a  single  element  in  Analytica.  Modules  are  also  used  to 
simplify  and  clarify  the  appearance  of  an  influence  diagram.  A  link 
appears  between  two  modules  if  a  node  in  one  module  is  dependent 
on  a  node  in  the  other  module. 

In  the  air-to-air  example,  each  module  comprises  the  nodes 
needed  to  compute  a  top-level  metric.  Figure  A.l  shows  the  modules 
in  the  top  level  of  the  air-to-air  model  (and  hence  the  metrics  com¬ 
puted  in  the  model).  The  figure  also  shows  the  overall  dependencies 
between  nodes  in  the  modules  (and  hence  the  dependencies  between 
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the  metrics).  The  top-level  diagram  closely  matches  the  top  level  of 
the  NCO  Conceptual  Framework,  with  the  exception  that  quality  of 
interactions,  shared  sense-making,  shared  decisions,  synchronization 
metrics,  and  degree  of  effectiveness  are  missing  from  the  diagram. 
This  set  of  metrics  is  not  incorporated  in  the  air-to-air  case  study, 
except  for  the  degree  of  effectiveness,  which  is  obtained  from  the 
JTIDS  Special  Operations  Project  results. 

Each  module  contains  a  number  of  subsidiary  nodes  used  to 
structure  the  subsidiary  metrics  for  each  top-level  metric.  Figure  A.2 
describes  the  types  of  nodes  used  in  the  Analytica  model.  The  air-to- 
air  model  employed  in  this  report  uses  decision,  constant,  variable, 
and  objective  nodes.  The  chance  node  is  not  used  because  we  have 
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Figure  A. 2 
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chosen  to  use  a  deterministic  model  based  on  probabilities.  The 
module  nodes  organize  what  would  otherwise  be  a  very  complicated 
influence  diagram  into  a  manageable  number  of  small  subinfluence 
diagrams.  The  model  uses  index  nodes  to  define  the  dimensions  used 
in  calculating  the  metrics. 

A  main  theme  with  Analytica  models,  and  the  air-to-air  exam¬ 
ple,  is  that  each  metric  is  a  multidimensional  array,  not  a  single  scalar. 
Each  entry  in  the  array  gives  a  metric  score  for  a  set  of  dimensional 
indices.  Array  dimensions  commonly  used  include  the  user/node 
(AWACS  or  one  of  the  four  F- 1 5s),  the  information  element  received 
by  the  user,  and  the  level  of  service  used  to  receive  the  information 
item.  For  the  latter,  “level  of  service”  can  be  either  a  network  service 
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(e.g.,  data  or  voice),  or  a  performance  band  (e.g.,  item  received  within 
less  than  one  second  or  within  one  to  ten  seconds  of  its  creation). 

To  compute  the  overall  metrics  shown  in  the  body  of  the  report, 
Analytica  allows  the  user  to  average  across  the  dimensions.  Impor¬ 
tantly,  the  dependencies  between  metrics  are  based  on  the  full  multi¬ 
dimensional  arrays,  not  on  the  summary  averages. 

By  convention,  we  use  boldface  to  refer  to  an  Analytica  array 
and  italics  to  refer  to  an  array  dimension. 

As  noted,  the  index  nodes  define  the  dimensions  used  in  the 
arrays,  as  well  as  the  entries  used  as  index  points  for  each  dimension. 
The  air-to-air  model  has  eight  dimensions  that  are  commonly  used  in 
the  Analytica  arrays.  Figure  A.3  shows  the  corresponding  index  nodes 
and  lists  the  entries  for  the  “Nodes”  dimension. 

The  dimensions  (and  their  constituent  entries)  are  as  follows: 

•  Data  objects  refers  to  the  17  tracks  (the  sensed  tracks)  detected 
by  the  Blue  aircraft  in  the  model.  It  includes  the  eight  tracks 
detected  by  AWACS  (four  Red  fighters,  four  Blue  fighters) ,  the 


Figure  A.3 

Air-to-Air  Model  Array  Dimensions 
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two  Red  tracks  detected  by  Blue  1 1 ,  the  two  Red  tracks  detected 
by  Blue  12,  and  the  track  each  Blue  plane  makes  of  itself  (five 
total) . 

•  Net  type  refers  to  the  two  types  of  networks  tested  in  the  air-to- 
air  case  study,  voice  only  and  Link  16. 

•  Time-to-use  levels  refers  to  the  two  timeliness  performance  bands 
used  throughout  the  air-to-air  case  study.  These  include  the  less- 
than-one-second  timeliness  band  (here  referred  to  as  “Target¬ 
ing”)  and  the  one-  to  ten-second  timeliness  band  (here  referred 
to  as  “Cue  Only”). 

•  Nodes,  as  shown  in  Figure  A. 3,  refers  to  the  five  Blue  aircraft 
(the  network  “nodes”  in  this  case  study),  including  AWACS  and 
the  four  Blue  fighters. 

•  Essential  elements  of  information  (EEIs)  refers  to  the  nine  unique 
tracks  in  the  model,  including  AWACS,  the  four  Blue  fighters, 
and  the  four  Red  fighters. 

•  Aircrafi  type  refers  to  whether  an  airplane  is  Red  or  Blue. 

•  Information  quality  metrics  refers  to  the  four  common  quality  of 
information  metrics  used  throughout  the  model  (detection 
completeness,  ID  correctness,  location  correctness,  and  velocity 
correctness).  To  simplify  the  analysis,  the  Analytica  dimension 
actually  contains  only  two  entries:  “ID”  and  “location.”  This 
simplification  is  valid  because,  as  noted  in  the  briefing,  detection 
completeness  and  ID  correctness  always  have  the  same  value  and 
location  correctness  and  velocity  correctness  always  have  the 
same  value. 

•  Decision  types  refers  to  the  four  advanced  types  of  tactical  deci¬ 
sions  enabled  by  individual  awareness  (accelerated  engagements, 
combatant  wingman,  advance  vectoring,  and  cooperative  forma¬ 
tions),  plus  a  tactic  to  represent  baseline  decisions  (those  tradi¬ 
tionally  enabled  by  voice-only  communications  networks). 

In  Analytica,  decision  and  constant  nodes  allow  users  to  declare 
arrays  of  inputs.  As  suggested  by  the  names,  constant  nodes  allow 
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users  to  input  an  array  that  remains  constant  for  all  experiments  in 
the  model  and  decision  nodes  allow  arrays  that  represent  experimen¬ 
tal  choices — for  example,  whether  voice-only  or  Link  16  is  chosen. 

Figure  A .4,  for  example,  shows  the  input  array  to  the  “mode 
speed”  decision  node  (an  input  to  the  quality  of  service  metric  node). 
This  is  a  two-dimensional  array,  consisting  of  the  net  type  and  mode 
speed  dimensions.  The  former  is  one  of  the  common  dimensions 
shown  in  Figure  A. 3  and  represents  the  choice  between  the  voice-only 
and  Link  16  architectures.  The  latter  is  a  self-titled  dimension  that 
allows  for  sensitivity  analysis.  In  this  Analytica  model,  we  have  three 
sensitivity  analysis  settings,  “low,  mid,  and  high.”  The  “mid”  setting 
is  used  to  compute  the  results  shown  in  the  paper  and  is  based  on 
real-world  times  to  transmit  tracks  over  voice-only  and  Link  16 
channels.  For  comparison,  the  “low”  setting  describes  networks  in 
which  the  transmission  speeds  are  much  slower  than  in  reality,  and 
the  “high”  setting  describes  the  reverse.  The  array  entries  show  the 
number  of  tracks  that  can  be  transmitted  per  second,  given  a  sensitiv¬ 
ity  setting  and  a  network  type  (voice-only  or  Link  1 6) . 

Variable,  chance,  and  objective  nodes  in  Analytica  contain  a 
function  that  computes  a  new  array  using  arrays  from  other  nodes  as 

Figure  A.4 
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arguments.  Each  node  contains  only  a  single  function,  resembling  a 
function  in  a  spreadsheet  cell.  However,  the  function  can  be  a  com¬ 
plicated  set  of  nested  conditional  expressions,  such  as  those  allowed 
by  Excel.  There  are  several  major  conventions  for  array  functions: 

•  Formulas  use  abbreviated  notation  to  describe  operations  com¬ 
bining  arrays,  e.g.  Array3  =  Opl  (Arrayl,  Array2  .  .  .  ). 

•  Some  array  operations  are  written  as  conventional  mathematical 
expressions — e.g.  Array  l*Array2. 

•  Special  operations  aggregate  a  matrix  over  one  of  its  dimen¬ 
sions — e.g.  sum  (Arrayl,  index  1)  replaces  the  index  1  dimension 
entries  with  the  sums  of  the  entries  along  that  dimension. 

•  From  a  mathematical  perspective,  Analytica  uses  array  opera¬ 
tions,  not  matrix  operations.  Thus,  if  Arrayl  and  Array2  have 
the  same  dimensions  (and  recall,  by  definition,  each  dimension 
has  a  unique  set  of  entries  associated  with  it),  an  array  operation 
is  performed  on  each  corresponding  pair  of  Arrayl  andArray2 
entries — e.g.,  each  entry  in  Array l*Array2  is  just  the  pairwise 
multiplication  of  the  corresponding  entries  in  Arrayl  and 
Array2. 

•  Suppose  Arrayl  and  Array2  have  different  dimensions.  Then, 
when  computing  the  results  of  matrix  operations,  Analytica  first 
expands  the  arrays  so  that  each  has  the  union  of  dimensions 
from  Arrayl  and  Array2.  The  expansion  involves  copying  the 
existing  array  for  each  entry  along  an  extra  dimension.  Thus,  if 
Arrayl  has  three  unique  dimensions,  and  Array2  has  two  unique 
dimensions,  Array  l*Array2  will  have  five  dimensions. 


Force 

We  now  consider  the  specifics  of  the  air-to-air  Analytica  model, 
beginning  with  the  force  module.  This  module  contains  a  single 
node — the  data  placement  constant  node.  This  node  is  an  input  array 
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that  indicates  which  Blue  aircraft  have  which  of  the  17  sensed  tracks 
in  the  model.  Each  sensed  track  is  assigned  uniquely  to  its  corre¬ 
sponding  Blue  aircraft.  The  node’s  array  has  values  of  one  if  the  cor¬ 
responding  Blue  plane  senses  the  track  organically,  and  zero  if  not.  It 
is  used  in  a  variety  of  the  calculations  to  determine  the  information 
organic  to  each  node.  Figure  A. 5  shows  the  entries  of  the  data  place¬ 
ment  node  for  this  case  study. 


Quality  of  Organic  Information 

The  quality  of  organic  information  subnetwork  has  two  nodes,  as 
shown  in  Figure  A.6.  Quality  of  organic  data  lists  the  initial  quality  of 
each  of  the  seventeen  sensed  tracks,  as  measured  by  whether  they  fall 
into  the  less-than-one-second  band  or  one-  to  ten-second  band. 
(Information  that  falls  in  the  less-than-one-second  band  is  also  placed 
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Figure  A. 6 
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in  the  one-  to  ten-second  band  to  facilitate  later  analysis).  As  previ¬ 
ously  discussed,  the  probabilities  that  tracks  fall  into  timeliness  bands 
are  the  “premetrics”  used  in  calculating  the  four  quality  of  informa¬ 
tion  metrics  in  the  air-to-air  case  study.  Thus,  we  input  these  prob¬ 
abilities  rather  than  input  scores  for  the  four  quality  of  information 
metrics  directly. 

Organic  data  array,  also  shown  in  Figure  A. 6,  adds  an  extra 
dimension,  showing  which  nodes  have  which  tracks  and  to  what 
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timeliness  band,  organically.  This  extra  dimension  is  needed  for  the 
calculation  of  the  quality  of  individual  information  and  quality  of 
shared  information  metrics. 


Degree  of  Networking 

The  degree  of  networking  module  contains  five  nodes,  shown  in  Fig¬ 
ure  A.7.  Node  reach  determines  whether  each  node  can  connect  to 
the  “network”  via  voice-only  or  Link  16.  Reach  then  uses  the  node 
reach  and  data  placement  inputs  to  determine  whether  each  of  the  17 


Figure  A. 7 
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sensed  tracks  can  be  posted  to  the  network  (either  with  voice-only  or 
Link  16).  In  this  case  study,  reach  is  a  placebo,  as  all  Blue  aircraft  can 
post  to  the  network,  even  if  via  voice-only. 

Mode  speed  determines  the  rate  at  each  node  can  transmit 
sensed  tracks  via  voice-only  or  Link  16.  Three  levels  of  speed  for  each 
mode  (“low,”  “mid,”  “high”)  are  used  for  sensitivity  analysis;  the 
“mid”  speeds  reflect  the  actual  performance  characteristics  of  voice 
transmission  and  Link  16.  Mode  Lossage  is  the  average  amount  of 
data  lost  in  attempting  to  transmit  a  track;  this  is  set  at  30  percent  for 
voice-only  and  0  percent  for  Link  16.  Quality  of  service  then  uses  the 
mode  speed,  mode  lossage,  and  data  placement  inputs  to  determine 
the  rate  at  which  each  sensed  track  will  be  posted  and  retrieved  via 
voice-only  or  Link  16.  The  metric  generated  by  the  quality  of  service 
node  is  a  three-dimensional  array  giving  the  posting- and-retrieval  rate 
(in  updates  per  second)  along  three  dimensions:  the  sensed  track,  the 
network  type,  and  the  mode  speed. 

As  noted  earlier  in  this  appendix,  the  Analytica  model  versions 
of  reach  and  quality  of  service  are  a  bit  different  than  what  was  shown 
in  the  main  body  in  the  report.  The  metrics  in  the  report  directly  rep¬ 
resented  the  technical  characteristics  of  the  voice-only  and  Link  16 
networks — these  become  the  node  reach,  mode  lossage,  and  mode 
speed  input  nodes  in  the  Analytica  Model.  In  the  Analytica  model, 
reach  and  quality  of  service  actually  calculate  “premetrics”  needed  by 
degree  of  information  share-ability,  calculating  whether  each  of  the 
17  sensed  tracks  can  be  posted  to  and  retrieved  from  the  network 
(reach)  and  the  rate  at  which  posting  and  retrieving  occurs  (quality  of 
service). 


Degree  of  Information  "Share-Ability" 

The  degree  of  information  share-ability  subnetwork  has  two  major 
submetrics  that  were  used  in  the  case  study:  quantity  of  data  posted 
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and  quantity  of  data  retrieved.  Figure  A.8  describes  the  subtree  com¬ 
puting  quantity  of  data  posted. 

The  metric  node  (“quantity  of  data  posted”)  depends  on  the 
following  input  nodes: 

•  Reach  determines  whether  each  node  can  connect  to  the  “net¬ 
work”  via  voice-only  or  Link  16. 

•  Quality  of  service  determines  the  rate  at  which  sensed  tracks  will 
be  posted  and  retrieved  via  voice-only  or  Link  16.  Because  in 
this  case  we  are  only  interested  in  the  rate  at  which  sensed  tracks 
will  be  posted,  we  subtract  the  mode  lossage  input.  Mode  los- 
sage  only  applies  to  retrieved  information. 

Figure  A.8 
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•  Node  priority  describes  the  percentage  of  the  network’s  band¬ 
width  (voice-only  or  Link  16)  dedicated  to  each  of  the  five  Blue 
planes  (nodes) .  In  this  model,  we  assume  that  AWACS  has  half 
the  available  bandwidth,  and  each  of  the  four  fighters  has  one- 
eighth  of  the  bandwidth  apiece. 

•  Capacity  share  is  similar  to  node  priority  but  with  the  percent¬ 
age  of  bandwidth  mapped  to  each  of  the  17  sensed  tracks.  The 
latter  is  an  intermediate  node  needed  to  calculate  the  quantity  of 
data  posted;  it  does  not  add  any  new  information  to  the  model. 

•  Data  priority  lists  the  percentage  of  a  node’s  allocated  band¬ 
width  devoted  to  updating  each  sensed  track.  In  this  model,  Red 
tracks  are  assumed  to  have  twice  the  allocation  of  Blue  tracks. 
Within  Red  and  Blue  categories,  each  track  gets  equal  allocation. 

•  Time-to-use  constraints  defines  the  maximum  time  allowed  to 
transmit  a  track  update  in  order  for  the  update  to  quality  for  a 
certain  timeliness  performance  band  less  than  one  second  or  one 
to  ten  seconds).  Consistent  with  the  bands,  the  maximum  times 
are  one  second  and  ten  seconds,  respectively. 

Quantity  of  data  posted  then  computes  the  total  expected  frac¬ 
tion  of  sensed  track  updates  that  will  be  transmitted  within  the  time- 
to-use  levels.  The  final  output  of  the  computation  is  a  four  dimen¬ 
sional  array  that,  given  a  sensed  track,  a  type  of  network,  a  posting 
rate  within  that  network  type,  and  a  timeliness  performance  band, 
gives  the  probability  that  the  sensed  track  has  been  posted  to  the  net¬ 
work  within  the  specified  timeliness  band. 

Figure  A.9  describes  the  subtree  used  to  calculate  quantity  of 
data  retrieved. 

This  metric  is  a  function  of  quantity  of  data  posted  plus  several 
other  nodes,  including  the  following: 

•  Mode  lossage  is  added  back  as  an  input.  It  lists  the  probability 
that  a  Blue  aircraft  received  the  sensed  track  update  correctly, 
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Figure  A. 9 
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assuming  it  was  properly  posted  to  the  network.  As  in  the  brief¬ 
ing,  this  probability  is  100  percent  for  Link  16  and  70  percent 
for  voice-only. 

•  Intent  to  retrieve  lists  the  probability  that  the  Blue  aircraft  does, 
in  fact,  attempt  to  retrieve  the  sensed  track  update.  This  is  a  pla¬ 
cebo  in  the  current  case  study  because  all  planes  are  listening 
continually  to  the  same  voice  channel  (and  receiving  track  data 
on  the  same  Link  16  subnetwork,  if  applicable). 

Quantity  of  data  retrieved  then  computes  a  five-dimensional 
array,  each  entry  of  which  describes  the  probability  that  a  Blue  air¬ 
craft  has  received  one  of  the  17  tracks  correctly  within  the  most 
recent  time-to-use  constraint  (less  than  one  second  or  one  to  ten  sec- 
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onds),  given  a  type  of  installed  network  (voice-only  or  Link  16)  and 
mode  speed  (low,  medium,  or  high). 

Quantity  of  data  retrieved  then  provides  input  to  quantity  of 
data  retrieved  statistics,  which  computes  the  aggregate  statistics  seen 
on  the  figures.  Note  that  quantity  of  data  retrieved  statistics  do  not 
provide  inputs  to  other  nodes. 


Quality  of  Individual  Information 

Figure  A.  10  describes  the  subtree  for  quality  of  individual  informa¬ 
tion. 


Figure  A. 10 

Quality  of  Individual  Information 
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The  first  node,  quality  of  individual  data,  computes  the  total 
likelihood  that  a  node  (one  of  the  Blue  aircraft)  has  received  one  of 
the  17  sensed  tracks  within  the  less-than-one-second  or  one-  to  ten- 
second  timeliness  bands,  either  organically  (from  quality  of  organic 
data)  or  retrieved  from  the  network  (from  quantity  of  retrieved  data, 
either  via  voice-only  or  Link  16).  It  assumes  the  implicit  fusion  of 
organic  network  and  network  data  sources.  However,  from  our 
knowledge  of  the  data  networks  and  information  requirements  in 
question,  this  is  a  reasonable  assumption  for  this  case  study. 

Recall  that  17  sensed  tracks  are  in  the  model,  but  only  nine 
unique  tracks  (the  five  Blue  aircraft  and  the  four  Red  aircraft)  because 
several  aircraft  are  tracked  multiple  times.  These  nine  tracks  are  the 
EEIs  for  this  report.  The  quality  of  individual  information  node  cal¬ 
culates  a  five-dimensional  array,  with  each  entry  giving  the  overall 
probability  that  a  Blue  aircraft  has  received  a  unique  track  in  a  par¬ 
ticular  timeliness  band,  by  receiving  at  least  one  of  the  corresponding 
sensed  tracks,  given  a  particular  network  type  and  mode  speed.  The 
data-to-information  constant  aids  this  calculation  by  mapping  each 
unique  track  to  a  corresponding  sensed  tracks.  Figure  A.ll  provides  a 
screenshot  of  a  quality  of  individual  information  array,  for  the  one-  to 
ten-second  timeliness  band,  voice-only,  and  medium-speed  network. 

The  quality  of  individual  information  node  assumes  the  implicit 
fusion  of  all  the  sensed  tracks  pertaining  to  the  same  unique  track. 
This  is  a  reasonable  assumption  because  a  single  track  on  an  airplane 
can  provide  all  the  required  information.  Note  that  we  also  assume 
that  no  difficulties  crop  up  in  converting  this  information  into  indi¬ 
vidual  awareness.  For  example,  we  assume  that  pilots  can  distinguish 
between  two  tracks  for  one  entity  and  tracks  for  multiple  entities 
based  on  standard  procedures,  etc.  For  voice  networks,  pilots  report 
giving  first  precedence  to  organically  sensed  tracks,  then  turn  to 
voice-reported  tracks.  For  Link  16  networks,  Blue  tracks  are  fused, 
and  tracks  of  the  same  plane  detected  via  F-15  radars  will  be  super¬ 
imposed.  AWACS  tracks,  on  a  ten-second  delay  by  definition,  appear 


Analytica  Model  for  Air-to-Air  Combat  Example  99 


Figure  A. 11 

A  Quality  of  Individual  Information  Array 
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as  a  different  symbol,  which  pilots  ignore  if  the  AWACS-tracked 
planes  are  detected  with  an  F- 1 5  radar. 

Quality  of  individual  information  summary  converts  quality  of 
individual  information’s  track  receipt  probabilities  into  the  four 
quality  of  information  metrics  scores  and  then  aggregates  these  scores 
into  reflections  of  what  each  node  receives  on  the  Blue  airplanes  and 
Red  airplanes  as  a  whole.  EEI-Red-or-Blue  assists  with  the  aggrega¬ 
tion  by  declaring  whether  particular  aircraft  are  Red  or  Blue.  Time- 
to-use  weights  assists  with  the  calculation  by  declaring  the  relative 
value  of  received  tracks  in  the  less-than-one-second  band  and  one-  to 
ten-second  band  for  each  of  the  four  quality  of  information  metrics. 
(Currently,  information  in  the  one-  to  ten-second  band  is  declared  to 
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have  25  percent  of  the  value  of  information  in  the  less-than-one- 
second  band  for  the  location  and  velocity  correctness  scores.) 


Degree  of  Shared  Information 

Figure  A.  12  describes  the  subtree  generating  quality  of  shared  infor¬ 
mation.  In  this  report,  we  are  interested  in  comparing  what  tracks  can 
be  shared  over  the  voice-only  and  Link  16  networks,  so  we  define 
“shared  information”  to  be  “information  transmitted  over  the  net¬ 
work  (voice-only  or  Link  16).”  Thus,  the  degree  of  shared  informa¬ 
tion  metrics  are  computed  with  respect  to  the  organically  collected 
information  that  could  be  shared  over  the  network.  In  comparison, 

Figure  A. 12 

Degree  of  Shared  Information 
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the  quality  of  individual  information  metrics  were  computed  with 
respect  to  all  the  relevant  information  in  the  battlespace  (all  nine  air¬ 
craft  tracks,  ideally  in  the  less-than-one-second  band). 

The  nodes  and  node  formulas  in  this  subtree  closely  resemble 
those  for  quality  of  individual  information.  There  are  two  differences. 
The  first  is  in  the  quality  of  shared  data  variable;  this  node  strictly 
computes  the  likelihood  that  a  node  has  received  one  of  the  17  sensed 
tracks  within  the  less-than-one-second  or  one-  to  ten-second  timeli¬ 
ness  bands  from  the  network  (voice-only  or  Link  16),  not  organically. 

The  second  is  the  shared  information  time-to-use  weights  con¬ 
stant,  employed  to  compute  the  degree  of  shared  information  sum¬ 
mary  statistics.  In  comparison  to  the  time-to-use  weights  constant  in 
individual  information,  this  constant  compensates  for  the  fact  that 
some  tracks  (Red  3  and  Red  4)  are  not  tracked  within  the  less-than- 
one-second  timeliness  band  by  any  of  the  Blue  planes.  The  compen¬ 
sated  weights  avoid  mistakenly  penalizing  the  degree  of  shared  infor¬ 
mation  for  not  sharing  real-time  information  about  Red  3  and  Red  4 
that  does  not  exist  organically. 


Quality  of  Individual  Awareness 

Figure  A.  13  describes  the  subtree  used  to  calculate  quality  of  individ¬ 
ual  awareness.  The  first  node,  so  named,  models  awareness  by  com¬ 
puting  the  likelihood  that  a  particular  pilot  “remembers”  a  particular 
EEI  based  on  previous  updates.  We  assume  a  50  percent  chance  of 
“forgetting”  an  EEI  in  each  successive  ten-second  interval  from  the 
period  when  it  was  first  received  (the  50  percent  is  a  parameter 
declared  in  retention  rate).  Here,  “forgetting”  means  either  that  the 
pilot  no  longer  remembers  the  update  or  that  the  tracked  plane  has 
moved  far  enough  that  the  update  is  no  longer  valid.  This  “memory” 
model  of  awareness  only  applies  to  information  items  in  the  one-  to 
ten-second  band  that  are  used  for  cuing  purposes.  It  does  not  apply  to 
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Figure  A. 13 

Quality  of  Individual  Awareness 
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information  items  in  the  less-than-one-second  timeliness  band. 
(However,  recall  that,  when  a  less-than-one-second  track  is  received, 
it  is  modeled  in  both  the  less-than-one-second  band  and  the  one-  to 
ten-second  band).  Information  in  the  less-than-one-second  timeliness 
bank  is  used  for  precision  maneuver  and  targeting  and  must  be 
updated  constantly. 

The  node  quality  of  individual  awareness  summary  computes 
aggregate  quality  of  awareness  metrics.  It  is  identical  in  nature  to  the 
previously  discussed  quality  of  individual  information  summary  node. 
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Quality  of  Individual  Decisionmaking 

Finally,  Figure  A.  14  describes  the  subtree  used  to  calculate  quality  of 
individual  decisionmaking,  the  penultimate  metric  in  the  air-to-air 
case  study.  (Recall  that  force  effectiveness  metrics  come  directly  from 
the  JTIDS  Operational  Special  Project  study.)  The  self-named  node 
computes  an  array  listing  whether  the  quality  of  individual  awareness 
is  sufficient  to  use  each  of  the  four  advanced  tactics  previously  dis¬ 
cussed  in  this  report. 

The  two  decision  nodes  declare  the  awareness  requirements 
needed  to  use  air-to-air  tactics  effectively.  “Requirements  for  decision 
types”  declares  which  EEIs  need  to  be  “known”  and  what  timeliness 


Figure  A. 14 

Quality  of  Individual  Decisions 
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bands  (less  than  one  second  or  one  to  ten  seconds)  are  needed  to  sup¬ 
port  improved  tactics  execution,  in  accordance  with  the  requirements 
shown  in  Table  10.1.  “Global  decision  requirements”  sets  the  likeli¬ 
hood  of  awareness  required  for  the  EEI  to  be  “known;”  by  default, 
this  parameter  is  set  at  90  percent.  The  quality  of  individual  deci¬ 
sionmaking  then  computes  an  array  of  “Is”  and  “Os”  by  performing 
checks  to  see  whether  all  the  awareness  conditions  for  a  Blue  aircraft 
to  run  an  advanced  tactic  have  been  met. 
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